In the two previous post we have seen how disk IO and network IO affects our ETLs. For both use cases we have seen several techniques that could be used to improve drastically performance and drive to an efficient resource usage:
Avoid IO disk at all.
Use buff/cache properly if IO disk couldn’t be avoided.
Optimize data download by choosing the right file format, use the Keep-Alive properly and parallelize network operations.
In this post we are going to put together network and processing operations to see the improvement in a complete workflow.
In the previous post I have focused in avoiding as much as possible IO on disk and if that was not possible using buff/cache as much as possible by grouping in time IO operations. This approach can make our ETL processes run X times faster. In the two examples the numbers where:
Avoiding IO at all was 11,3 times faster
Using buff/cache was almost 4 times faster
All the examples used a dataset already in the disk so no real network operation occurred. In this post I am going to focus on network operation using again GNU parallel.
Several months ago I finally got the (WOL) Wake On Lan feature of my RTL8111/8168B NIC card working. The problem was that a new driver (other than the provided by Debian) and a special PCI configuration was needed.
The other problem I had to deal with was the ADSL Router (Comtrend HG532c, The one provided by the Spanish ISP Jazztel) configuration:
Open the required port: This was an easy one just opening the 7 a 9 port and forwarding them to the server we want to WOL from the internet
Make the router remember the server’s tuple MAC/IP address. That was easy too, but some manual work was needed as when router is restarted the ARP table is flushed. 🙁
In my current job I had to change recently some configuration and restart more than 600 IP phones. To perform such titanic task I created a quick and dirty script using expect. It worked like a charm and made me think about automatize the way I set the ARP table in my Comtrend HG532c ADSL router.
Usamos cookies para ofrecerte la mejor experiencia en nuestra web.
Haciendo clic en “Aceptar” das tu consetimiento para usar estas cookies.
Puedes realizar un consentimiento pormenorizado en \"Ajustes\".
Configurar y más información