A Quick and Neat :) Orchestrator using GNU Parallel

Sometimes you have to deal with servers that you don’t know anything about:

  • You are a short temp IT consultant with not previous knowledge on the environment.
  • The CMDB is out of order.
  • You are on a DR situation.
  • Or simply the main administrator is not there.

And you need:

  • Run commands in parallel
  • Get info from many servers at a time
  • Troubleshoot DNS problems
  • Check how many servers are up and running

On my systems I use two orchestrators: MCollective and SaltStack (configured automatically using puppet) that fulfill my needs. But let’s see a quick way to have an orchestrator in a rapid manner.

Basic Orchestrator architecture

Both tools have a “reverse” client/server architecture where the nodes are connected to the server launching the commands.

SaltStack can run also in a masterless way too using the salt-ssh package:

  1. This particular package provides the salt ssh controller. It is able to run salt modules and states on remote hosts via ssh. No minion or other salt specific software needs to be installed
  2.  on the remote host.

What an orchestrator can do?

Let’s see some examples with MCollective or SaltStack

See how many of your servers are up and running right now.

With both tools you launch a ping-like command.

  • Mcollective
<code class="console"><span class="gp">%</span> mco ping
<span class="go">server1                                    time=126.19 ms</span>
<span class="go">server2                                    time=132.79 ms</span>
<span class="go">server3                                    time=133.57 ms</span>
<span class="go">---- ping statistics ----</span>
<span class="go">25 replies max: 305.58 min: 57.50 avg: 113.16</span></code>
  • SaltStack:
  1. salt '*' test.ping
  2. server1:
  3.     True
  4. server2:
  5.     True
  6. server3:
  7.     True
  8. server4:
  9.     Minion did not return. [No response]

Stop/Start/Status services on many servers:

  • Mcollective:
  1. mco service status ssh
  2.  
  3. * [ ==========================================> ] 1 / 3
  4.  
  5. server1: running
  6. server2: running
  7. server3: running
  8.  
  9. Summary of Service Status:
  10.  
  11.    running = 3
  12.  
  13. Finished processing 3 / 3 hosts in 116.32 ms
  • SaltStack:
  1. salt '*' cmd.run '/etc/init.d/ssh status'
  2. server1:
  3.     sshd is running.
  4. server2:
  5.     sshd is running.
  6. server3:
  7.     sshd is running.

Other stuff:

  • Mcollective (list from Bluemalkin’s blog) :
    • puppet: manage Puppet agents (run a test, enable / disable, get statistics etc…)
    • package: install, uninstall a package
    • apt: upgrade packages, list number of available upgrades
    • service: start, stop, restart a service
    • nettest: check ping and telnet connectivity
    • filemgr: touch, delete files
    • process: list, kill process
    • nrpe: run nrpe commands (check_load, check_disks, check_swap)
      and more
  • SaltStack:

With SaltStack you can run almost every command-line tool:

  1. salt '*' cmd.run 'uname -an'
  2. server1:
  3.     Linux server1 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux
  4. server2:
  5.     Linux server2 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) x86_64 GNU/Linux
  6. server3:
  7.     Linux server3 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) x86_64 GNU/Linux

But you don’t have and orchestrator configured right now

And you need one. Let’s have a look to GNU parallel:

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

Usually it is used to spread high resources (CPU/Memory/etc) demanding task across multiple servers. But GNU parallel can be used as an orchestrator too in a similar way as MCollective and Saltstack.

Installation:

Usually it already exists on the main Linux official repositories. It it doesn’t exist for yours, it can be downloaded from http://ftp.gnu.org/gnu/parallel/

  1. apt-get install parallel

Configuration:

In order to launch password less commands to remote servers using ssh public-key you need the private key on your user’s home and the public key on the .ssh/authorized_keys. In a normal infrastructure this should already exist.

As there is a lot of literature about it I am not going there again: passwordless ssh setup

Also you need a list of servers to launch commands to:

  1. cat > server.list
  2. server1
  3. server2
  4. server3

The command

This is the magic command that will do the trick:

  1. parallel --tag --nonall -j2 -k --slf server.list 'host google.es'

This command is going to launch the ‘host google.es’ command in the servers that are declared on the server.list file previously created.

–tag will tag the output with the server’s name the lines belong to.

–nonall this is useful for running the same command (e.g. uptime) on a list of servers.

-j2 the number of jobs to be launched in parallel.

-k Keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes.

-slf file name with the list of the servers

And finally the output is:

  1. server1        google.es has address 74.125.136.94
  2. server1        google.es has IPv6 address 2a00:1450:4013:c01::5e
  3. server1        google.es mail is handled by 30 alt2.aspmx.l.google.com.
  4. server1        google.es mail is handled by 10 aspmx.l.google.com.
  5. server1        google.es mail is handled by 40 alt3.aspmx.l.google.com.
  6. server1        google.es mail is handled by 50 alt4.aspmx.l.google.com.
  7. server1        google.es mail is handled by 20 alt1.aspmx.l.google.com.
  8. server2        google.es has address 216.58.211.227
  9. server2        google.es has IPv6 address 2a00:1450:4003:805::2003
  10. server2        google.es mail is handled by 50 alt4.aspmx.l.google.com.
  11. server2        google.es mail is handled by 10 aspmx.l.google.com.
  12. server2        google.es mail is handled by 20 alt1.aspmx.l.google.com.
  13. server2        google.es mail is handled by 40 alt3.aspmx.l.google.com.
  14. server2        google.es mail is handled by 30 alt2.aspmx.l.google.com.
  15. server3        google.es has address 216.58.211.227
  16. server3        google.es has IPv6 address 2a00:1450:4003:805::2003
  17. server3        google.es mail is handled by 10 aspmx.l.google.com.
  18. server3        google.es mail is handled by 50 alt4.aspmx.l.google.com.
  19. server3        google.es mail is handled by 20 alt1.aspmx.l.google.com.
  20. server3        google.es mail is handled by 40 alt3.aspmx.l.google.com.
  21. server3        google.es mail is handled by 30 alt2.aspmx.l.google.com.

As server1 is located on a different place with different DNS configuration than server2 and server3, the output is different.

The command can be used for example to troubleshoot DNS replication problems between servers.

Other uses cases:

You can launch more or less the same commands we have seen before with SaltStack and Mcollective and any other that you can imagine. Yeah! Here your imagination is the limit:

  • Populate your CMDB with the content of the /etc/resolv.conf of all your servers in a matter of seconds
  • Check the ldap configuration of all you servers
  • An “orchestrated grep” or “orchestrated log parser” for all your servers
  • Install packages
  • Patch servers
  • etc.

I will add more examples in a future. Here you are some examples:

Check connectivity with a local mysql database

  1. parallel --tag --nonall -j2 -k --slf server.list  'telnet databaseserver 3306'                                                                   
  2. server1        Trying 127.0.0.1...
  3. server1        Connected to localhost.
  4. server1        Escape character is '^]'.
  5. server1        S
  6. server1        5.5.43-0+deb7u1Ȉ%#}o5YA�Sy*a*M(zRi9!mysql_native_passwordConnection closed by foreign host.
  7. server2        Trying ::1...
  8. server2        Trying 127.0.0.1...
  9. server2        telnet: Unable to connect to remote host: Connection refused
  10. server3        Trying ::1...
  11. server3        Connected to localhost.
  12. server3        Escape character is '^]'.
  13. server3        S
  14. server3        5.5.43-0+deb7u1LjzsS$XnB�Q\Z0|Dz\@hmgmysql_native_passwordConnection closed by foreign host.

In this case we can see that servers 1 a 3 have a mysql database installed and the mysql daemon is listening on the standard port 3306.

Check pings:

  1. parallel --tag --nonall -j2 -k --slf server.list  'ping -c 1 google.es'                                                                     
  2. server1        PING google.es (74.125.136.94) 56(84) bytes of data.
  3. server1        64 bytes from ea-in-f94.1e100.net (74.125.136.94): icmp_req=1 ttl=47 time=5.68 ms
  4. server1
  5. server1        --- google.es ping statistics ---
  6. server1        1 packets transmitted, 1 received, 0% packet loss, time 0ms
  7. server1        rtt min/avg/max/mdev = 5.684/5.684/5.684/0.000 ms
  8. server2        PING google.es (216.58.211.227) 56(84) bytes of data.
  9. server2        64 bytes from mad01s24-in-f3.1e100.net (216.58.211.227): icmp_seq=1 ttl=45 time=156 ms
  10. server2
  11. server2        --- google.es ping statistics ---
  12. server2        1 packets transmitted, 1 received, 0% packet loss, time 0ms
  13. server2        rtt min/avg/max/mdev = 156.197/156.197/156.197/0.000 ms
  14. server3  PING google.es (216.58.211.227) 56(84) bytes of data.
  15. server3  64 bytes from mad01s24-in-f3.1e100.net (216.58.211.227): icmp_seq=1 ttl=45 time=36.0 ms
  16. server3
  17. server3  --- google.es ping statistics ---
  18. server3  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  19. server3  rtt min/avg/max/mdev = 36.018/36.018/36.018/0.000 ms

All servers can ping google.es

A word of caution about orchestrators:

Remember what Uncle Ben says:

With great power comes great responsibility

Uncle Ben

Orchestrators are a very useful tools but they can do a lot of harm if they are not managed carefully. As you can launch an inappropriate command in many servers in parallel 🙁

With great power comes great responsibility

Share

2 thoughts on “A Quick and Neat :) Orchestrator using GNU Parallel

  1. Pingback: Optimizing long batch processes or ETL by using buff/cache properly | El Sotanillo de Juan Sierra Pons

Leave a Reply

Your email address will not be published. Required fields are marked *

 

This site uses Akismet to reduce spam. Learn how your comment data is processed.