Run comand in parellel in linux shell
Methodd 1 : use shell built-in “wait” to run command in parellel
j@ubuntu2004:~$ sleep 10 & time wait [1] 4564 [1]+ Done sleep 10 real 0m10.002s user 0m0.002s sys 0m0.000s j@ubuntu2004:~$
We can see command wait will be waiting for “sleep 10”, so we can run particular numbers of cmds in background ,then wait then run another particular number of command, like below
cmd1 & cmd2 & cmd3 & wait cmd4 & cmd5 & wait cmd6 &
A simple example below :
j@ubuntu2004:~/bin$ cat test.sh sleep 2 & sleep 2 & sleep 2 & wait sleep 2 & sleep 2 & sleep 2 & wait sleep 2 & sleep 2 & sleep 2 & wait j@ubuntu2004:~/bin$ time ./test.sh real 0m6.025s user 0m0.011s sys 0m0.029s
Method 2: use xargs to run cmds in parellel
-P max-procs, --max-procs=max-procs Run up to max-procs processes at a time; the default is 1.
Above is from manpage of command xargs
So we can run command in parellel using below format:
somecontent | xargs -n1 -P N command
Eg: suppose we want to ping multiple hosts in parellel, we can:
j@ubuntu2004:~$ cat hosts.txt host01 host02 host03 host04 host05 host06 host07 host08 host09 host10 j@ubuntu2004:~$ cat hosts.txt |xargs -n 1 -P 5 ping -c 1
ssh to multiple hosts in parellel (5 processes) to get “uptime”
j@ubuntu2004:~$ cat hosts.txt |xargs -n 1 -P 5 -I '{}' ssh '{}' uptime 09:57:46 up 2:09, 1 user, load average: 0.15, 0.36, 0.22 09:57:46 up 2:09, 1 user, load average: 0.15, 0.36, 0.22 09:57:46 up 2:09, 1 user, load average: 0.15, 0.36, 0.22 09:57:46 up 2:09, 1 user, load average: 0.15, 0.36, 0.22 09:57:46 up 2:09, 1 user, load average: 0.15, 0.36, 0.22 09:57:47 up 2:09, 1 user, load average: 2.06, 0.76, 0.35 09:57:47 up 2:09, 1 user, load average: 2.06, 0.76, 0.35 09:57:47 up 2:09, 1 user, load average: 2.06, 0.76, 0.35 09:57:47 up 2:09, 1 user, load average: 2.06, 0.76, 0.35 09:57:47 up 2:09, 1 user, load average: 2.06, 0.76, 0.35
-I replace-str Replace occurrences of replace-str in the initial-arguments with names read from standard input.
To discard “exited with status 255; aborting” error ,add “sh -c” and “ | true” |
cat hosts.txt |xargs -n 1 -P 5 -I '{}' sh -c "ssh '{}' uptime||true"