11 ms·

GNU Parallel

100 points by todd8 5y ago

ValtteriL 5y ago
Parallel has been (and still is) super useful and simple tool for speeding up all kinds of shell tasks during my career. Still remember the time I wasted days waiting for the completion of multiple serial tasks when it would have been an hour with parallel :-)
cb321 5y ago
With the default setup, GNU parallel seems to be about 1000x slower than it should be and 165x slower than serial xargs. On a 16 core/32 thread CPU running Linux 5.16 (with parallel-20220322, perl-5.34.1, ripgrep-13.0.0, findutils-4.9.0, grep-3.7): cd /dev/shm wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.2.tar.xz tar xpJf linux-5.17.2.tar.xz cd linux-5.17.2 rg -a --no-ignore --files | tr \\n \\0 > ../f tm="/usr/bin/time"; lb="--line-buffered" $tm rg -a --no-ignore -l foofoo # 130 milliseconds wall $tm xargs -0P16 grep $lb -l foofoo < ../f # 120 milliseconds wall $tm xargs -0 grep $lb -l foofoo < ../f # 800 milliseconds wall $tm parallel -0j16 grep -l foofoo < ../f # 132 *SECONDS* wall GNU parallel uses massive %CPU to make slow progress which also tends to confuse users new to parallelism in general. Hey, maybe there is a --dont-go-slow flag somewhere. I'd bet actual new users take a while to find it - maybe they never find it. There seems to at least some story of bad/confusing defaults here. A quick grep in the parallel package for benchmark only seems to indicate things that benefit from network asynchrony, not CPU parallelism. I tried to pick something anyone could reproduce, should they so desire. Cheers.
- cb321 5y ago
  Addendum: I guess parallel is not a "drop-in" for xargs by default, but more like xargs -n1, as has been mentioned elsethread. You need to use -X (that is a capital 'x' as lowercase 'x' means something else following the xargs CLI syntax, but not its semantics, LOL [1]). $tm parallel -X0j16 grep -l foofoo < ../f # 2.33 *SECONDS* wall So, GNU parallel is "only" 20X slower than it should be and "only" 3X slower than serial xargs on a 16 core box (on bare metal, not in the cloud, by the way). Still pretty awful unless there is another "--dont-burn-down-forests-for-electricity" flag I'm missing. [1] https://unix.stackexchange.com/questions/273170/gnu-parallel-excessively-slow https://unix.stackexchange.com/questions/273170/gnu-parallel...
- caro11ne 5y ago
  I find this scary: $ export LC_ALL=C $ $tm xargs -0P1 grep $lb t < ../f |sort |md5sum 7.05user 9.88system 0:24.53elapsed 69%CPU (0avgtext+0avgdata 2344maxresident)k 0inputs+0outputs (0major+4960minor)pagefaults 0swaps 8ef2c658a70bb38438e59421231246b9 - $ $tm xargs -0P16 grep $lb t < ../f |sort |md5sum 10.16user 36.62system 0:18.30elapsed 255%CPU (0avgtext+0avgdata 2332maxresident)k 0inputs+0outputs (0major+4980minor)pagefaults 0swaps c8ebf840e54ec8b5a49e159eda09e63f - $ $tm parallel -X0P16 grep $lb t < ../f |sort |md5sum 16.97user 33.94system 0:16.36elapsed 311%CPU (0avgtext+0avgdata 51624maxresident)k 0inputs+2069296outputs (0major+169409minor)pagefaults 0swaps 8ef2c658a70bb38438e59421231246b9 - It greps for lines containing t, sorts the lines and computes a hash. Note how "xargs -P16 grep" gives the wrong answer. The output from parallel matches exactly the lines from "xargs -P1". With "-k" the lines are even in the same order (sorting removed): $ $tm xargs -0P1 grep $lb t < ../f |md5sum 7.03user 9.30system 0:16.32elapsed 100%CPU (0avgtext+0avgdata 2332maxresident)k 0inputs+0outputs (0major+5023minor)pagefaults 0swaps d89b45188602c9bb08026dc2892cfa75 - $ $tm parallel -kX0P16 grep $lb t < ../f |md5sum 18.21user 36.03system 0:10.26elapsed 528%CPU (0avgtext+0avgdata 65396maxresident)k 0inputs+2069344outputs (0major+154929minor)pagefaults 0swaps d89b45188602c9bb08026dc2892cfa75 - I have not analyzed the output but I think the error is caused by the issue described here: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w... How anyone would ever use "xargs -P16 grep" is beyond me. I honestly do not care how fast I can get an answer, if I cannot trust the answer is correct. I can see someone claimed they could build a safe parallel grep, but seemed not to do so: https://news.ycombinator.com/item?id=30890780#30913304 https://news.ycombinator.com/item?id=30890780#30913304 It would have been interesting to see.
  - cb321 5y ago
    You are just moving goalposts from "grep -l" to "grep t". The "grep -l" should be reliable by virtue of line buffering and Linux kernel source path names being shorter than PIPE_BUF (which yes, you do have to know|check - much less to know than a >5000 line man page). While I could address the moved goalposts, I already mentioned xargs --process-slot-var [1] elsethread and, in my experience, goalpost movers are never satisfied. [1] https://unix.stackexchange.com/questions/449224/how-can-i-get-the-index-of-the-xargs-parallel-processor https://unix.stackexchange.com/questions/449224/how-can-i-ge...
dehrmann 5y ago
It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance future development and should feel free to use --will-cite in scripts. > If you do not want to help financing future development by letting other users see the citation notice or by paying, then please consider using another tool instead of GNU parallel. You can find some of the alternatives in man parallel_alternatives.
- stabbles 5y ago
  I don't understand how GNU finds this acceptable...
  - CaliforniaKarl 5y ago
    [edited for formatting and to fix Git link] According to https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/citation-notice-faq.txt https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... > == Is the citation notice compatible with GPLv3? == > Yes. The wording has been cleared by Richard M. Stallman to be compatible with GPLv3. This is because the citation notice is not part of the license, but part of academic tradition. > Therefore the notice is not adding a term that would require citation as mentioned on: https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation If you are of the view that clearance by RMS is clearance by FSF/GNU, then that's how they find it acceptable. If you take a different view, then the next part of that section applies: > If you disagree with Richard M. Stallman's interpretation and feel the citation notice does not adhere to GPLv3, you should treat the software as if it is not available under GPLv3. And since GPLv3 is the only thing that would give you the right to change it, you would not be allowed to change the software. There's also an interesting comparison to be made: > == How do I silence the citation notice? == > Run this once: > parallel --citation > It takes less than 10 seconds to do and is thus comparable to an 'OK. Do not show this again'-dialog box seen in LibreOffice, Firefox and similar programs.
deleted 5y ago
[deleted]
WhatIsDukkha 5y ago
This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... It has a lot of features that can feel excessive at first glance but if you have felt some pain in building jobs, most of it is pretty sensible and much better than rollyourown.
- cb321 5y ago
  The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.
  - ZoomZoomZoom 5y ago
    > Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Well, there was a Rust version with zero Perl, now unfortunately archived. It wasn't 100% on a par with the original and wasn't really finished. On the other hand, built easily for Windows and helped me on a few occasions. https://github.com/mmstick/parallel https://github.com/mmstick/parallel
    - cb321 5y ago
      Well, some archived project is not so great either. The core functionality is not even a 20 line bash script since bash grew wait -n, though: #!/bin/bash if [ "${1-0}" -lt 1 ]; then # No arg / arg not a number >= 1 echo "Usage: $0 <N>"; echo "reads cmds from stdin, running up to N at once." exit 1 fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1 while read cmd; do jobs > $TMP # jobs | wc -l does not work if [ $(wc -l < $TMP) -ge $1 ]; then wait -n # Wait for 1/more jobs to finish fi # Could accumulate total $? above, but would need to replace final wait. STRIPE_SEQ=$((STRIPE_SEQ + 1)) ( eval "$cmd" ) < /dev/null & # Run job in a subshell in bg done rm -f $TMP wait # Wait for all to finish and if bash ever grows some magic environment variable $NUM_BG_JOBS or you don't want auto-help or sequence numbers or etc. it can be even simpler.
dang 5y ago
Related: GNU Parallel 2018 - https://news.ycombinator.com/item?id=20726631 https://news.ycombinator.com/item?id=20726631 - Aug 2019 (68 comments) GNU Parallel Cheat Sheet [pdf] - https://news.ycombinator.com/item?id=19330356 https://news.ycombinator.com/item?id=19330356 - March 2019 (63 comments) GNU Parallel - https://news.ycombinator.com/item?id=13258142 https://news.ycombinator.com/item?id=13258142 - Dec 2016 (83 comments) GNU Parallel Tutorial - https://news.ycombinator.com/item?id=12943150 https://news.ycombinator.com/item?id=12943150 - Nov 2016 (65 comments) A Million Text Files and a Single Laptop - https://news.ycombinator.com/item?id=11248326 https://news.ycombinator.com/item?id=11248326 - March 2016 (27 comments) GNU Parallel – The command line power tool - https://news.ycombinator.com/item?id=6209767 https://news.ycombinator.com/item?id=6209767 - Aug 2013 (28 comments) GNU/Parallel changed my life - https://news.ycombinator.com/item?id=1894639 https://news.ycombinator.com/item?id=1894639 - Nov 2010 (8 comments) GNU Parallel - build and execute command lines from standard input in parallel - https://news.ycombinator.com/item?id=1801186 https://news.ycombinator.com/item?id=1801186 - Oct 2010 (36 comments)
arrakeen 5y ago
parallel is so useful and i use it multiple times daily. i wish its `:::` syntax was supported at the shell level so i could use it for every application
paskozdilar 5y ago
GNU Parallel is one of my favorite utilities of all time. I used to write complex and bug-ridden scripts that used bash job control to implement parallel execution of batch jobs. When I discovered GNU Parallel, I deleted them all and never looked back. Also, the documentation is awesome - there is a whole book [0] on GNU Parallel, and the manpage even links to a series of youtube videos [1] that explain how it works. [0] https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pdf https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pd... [1] http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
mekster 5y ago
If you want to execute commands on multiple remote hosts over SSH, just run a tmux session and launch a tab on each host in a loop and execute them. It's far easier to follow the output and individually deal with prompts.
- permalac 5y ago
  I do the same, I even have a command that tells me which tmux panel I'm in so I can ssh to many servers at once from tmux. However, when you just want to execute the same command : https://clustershell.readthedocs.io/en/latest/tools/clush.html https://clustershell.readthedocs.io/en/latest/tools/clush.ht...
- paskozdilar 5y ago
  True, but for a large number of hosts, going manually through prompts doesn't scale anymore. Sometimes you just want to check whether all hosts have the same configuration file, or something like that. That's where GNU Parallel can save your gluteus maximus: cat hosts.txt | parallel --quote --timeout=10 ssh {} 'echo {} $(md5sum ~/.config/file)' The above command would take a list of hostnames from the hosts.txt file, connect to each one, hash the config file and print out hostnames and hashes in one line per host.
  - mekster 5y ago
    Unless entire operations can be finished automatically, you could still do, command_to_test && exit in the swarm of tmux tabs and let only the anomalies stay open.
  - MainJane 5y ago
    cat hosts.txt | parallel --quote --timeout=10 ssh {} 'echo {} $(md5sum ~/.config/file)' Also try: parallel --slf hosts.txt --timeout=10 --nonall --tag md5sum .config/file
- ktm5j 5y ago
  I love tmux for this application, and typically I tend to prefer alternate solutions to using parallel.. but to be fair there are many more usecases for parallel than the one you describe here.
technofiend 5y ago
There's probably a more unix-centric / idiomatic way to do it, but using GNU Parallel along with Redis made automating tasks to process tens of thousands of systems extremely easy, repeatable, pausable, restartable, etc. And the fact you can do atomic operations on queues prevented any potential worker collisions.
winrid 5y ago
For those that don't know, it's written in Perl! Such a life saver of a tool. I used GNU Parallel to run a script locally that did what a big distributed system did, quicker and more reliably. It got to the point where people would just ask me to "run the thing" on my laptop instead of waiting for the cron. Mirror: https://github.com/martinda/gnu-parallel https://github.com/martinda/gnu-parallel
desktopninja 5y ago
Can't quite pin my resistance to parallel but I've enjoyed using this magic for many moons now: xargs -t -n 1 -P 3 -I TARGET_COMPUTER ssh -q TARGET_COMPUTER " sleep \$(shuf -i 1-9 | head -1) echo \$(hostname; date) echo \"===> \$(hostname) <===\" > ~/I_WAS_HERE " <<< " server-01 server-02 server-03 server-04 server-05 "
- caro11ne 5y ago
  I use: iwashere() { sleep $(shuf -i 1-9 | head -1) date echo "===> $(hostname) <===" > ~/I_WAS_HERE } env_parallel -Sserver-0{1..5} --tag --nonall iwashere I like that I can try out the function locally before running it remotely. I like that I do not have to give xargs a multiline argument. I am terrible at quoting inside quoting inside quoting, so I like that I can simply avoid the quoting. I also think my colleagues will find it easier to read (thus maintain).

100 points by todd8 5y ago

Parallel has been (and still is) super useful and simple tool for speeding up all kinds of shell tasks during my career. Still remember the time I wasted days waiting for the completion of multiple serial tasks when it would have been an hour with parallel :-)

cb321 5y ago

With the default setup, GNU parallel seems to be about 1000x slower than it should be and 165x slower than serial xargs. On a 16 core/32 thread CPU running Linux 5.16 (with parallel-20220322, perl-5.34.1, ripgrep-13.0.0, findutils-4.9.0, grep-3.7): cd /dev/shm wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.2.tar.xz tar xpJf linux-5.17.2.tar.xz cd linux-5.17.2 rg -a --no-ignore --files | tr \\n \\0 > ../f tm="/usr/bin/time"; lb="--line-buffered" $tm rg -a --no-ignore -l foofoo # 130 milliseconds wall $tm xargs -0P16 grep $lb -l foofoo < ../f # 120 milliseconds wall $tm xargs -0 grep $lb -l foofoo < ../f # 800 milliseconds wall $tm parallel -0j16 grep -l foofoo < ../f # 132 *SECONDS* wall GNU parallel uses massive %CPU to make slow progress which also tends to confuse users new to parallelism in general. Hey, maybe there is a --dont-go-slow flag somewhere. I'd bet actual new users take a while to find it - maybe they never find it. There seems to at least some story of bad/confusing defaults here. A quick grep in the parallel package for benchmark only seems to indicate things that benefit from network asynchrony, not CPU parallelism. I tried to pick something anyone could reproduce, should they so desire. Cheers.

cb321 5y ago
Addendum: I guess parallel is not a "drop-in" for xargs by default, but more like xargs -n1, as has been mentioned elsethread. You need to use -X (that is a capital 'x' as lowercase 'x' means something else following the xargs CLI syntax, but not its semantics, LOL [1]). $tm parallel -X0j16 grep -l foofoo < ../f # 2.33 *SECONDS* wall So, GNU parallel is "only" 20X slower than it should be and "only" 3X slower than serial xargs on a 16 core box (on bare metal, not in the cloud, by the way). Still pretty awful unless there is another "--dont-burn-down-forests-for-electricity" flag I'm missing. [1] https://unix.stackexchange.com/questions/273170/gnu-parallel-excessively-slow https://unix.stackexchange.com/questions/273170/gnu-parallel...
caro11ne 5y ago
I find this scary: $ export LC_ALL=C $ $tm xargs -0P1 grep $lb t < ../f |sort |md5sum 7.05user 9.88system 0:24.53elapsed 69%CPU (0avgtext+0avgdata 2344maxresident)k 0inputs+0outputs (0major+4960minor)pagefaults 0swaps 8ef2c658a70bb38438e59421231246b9 - $ $tm xargs -0P16 grep $lb t < ../f |sort |md5sum 10.16user 36.62system 0:18.30elapsed 255%CPU (0avgtext+0avgdata 2332maxresident)k 0inputs+0outputs (0major+4980minor)pagefaults 0swaps c8ebf840e54ec8b5a49e159eda09e63f - $ $tm parallel -X0P16 grep $lb t < ../f |sort |md5sum 16.97user 33.94system 0:16.36elapsed 311%CPU (0avgtext+0avgdata 51624maxresident)k 0inputs+2069296outputs (0major+169409minor)pagefaults 0swaps 8ef2c658a70bb38438e59421231246b9 - It greps for lines containing t, sorts the lines and computes a hash. Note how "xargs -P16 grep" gives the wrong answer. The output from parallel matches exactly the lines from "xargs -P1". With "-k" the lines are even in the same order (sorting removed): $ $tm xargs -0P1 grep $lb t < ../f |md5sum 7.03user 9.30system 0:16.32elapsed 100%CPU (0avgtext+0avgdata 2332maxresident)k 0inputs+0outputs (0major+5023minor)pagefaults 0swaps d89b45188602c9bb08026dc2892cfa75 - $ $tm parallel -kX0P16 grep $lb t < ../f |md5sum 18.21user 36.03system 0:10.26elapsed 528%CPU (0avgtext+0avgdata 65396maxresident)k 0inputs+2069344outputs (0major+154929minor)pagefaults 0swaps d89b45188602c9bb08026dc2892cfa75 - I have not analyzed the output but I think the error is caused by the issue described here: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w... How anyone would ever use "xargs -P16 grep" is beyond me. I honestly do not care how fast I can get an answer, if I cannot trust the answer is correct. I can see someone claimed they could build a safe parallel grep, but seemed not to do so: https://news.ycombinator.com/item?id=30890780#30913304 https://news.ycombinator.com/item?id=30890780#30913304 It would have been interesting to see.
- cb321 5y ago
  You are just moving goalposts from "grep -l" to "grep t". The "grep -l" should be reliable by virtue of line buffering and Linux kernel source path names being shorter than PIPE_BUF (which yes, you do have to know|check - much less to know than a >5000 line man page). While I could address the moved goalposts, I already mentioned xargs --process-slot-var [1] elsethread and, in my experience, goalpost movers are never satisfied. [1] https://unix.stackexchange.com/questions/449224/how-can-i-get-the-index-of-the-xargs-parallel-processor https://unix.stackexchange.com/questions/449224/how-can-i-ge...

dehrmann 5y ago

It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance future development and should feel free to use --will-cite in scripts. > If you do not want to help financing future development by letting other users see the citation notice or by paying, then please consider using another tool instead of GNU parallel. You can find some of the alternatives in man parallel_alternatives.

stabbles 5y ago
I don't understand how GNU finds this acceptable...
- CaliforniaKarl 5y ago
  [edited for formatting and to fix Git link] According to https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/citation-notice-faq.txt https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... > == Is the citation notice compatible with GPLv3? == > Yes. The wording has been cleared by Richard M. Stallman to be compatible with GPLv3. This is because the citation notice is not part of the license, but part of academic tradition. > Therefore the notice is not adding a term that would require citation as mentioned on: https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation If you are of the view that clearance by RMS is clearance by FSF/GNU, then that's how they find it acceptable. If you take a different view, then the next part of that section applies: > If you disagree with Richard M. Stallman's interpretation and feel the citation notice does not adhere to GPLv3, you should treat the software as if it is not available under GPLv3. And since GPLv3 is the only thing that would give you the right to change it, you would not be allowed to change the software. There's also an interesting comparison to be made: > == How do I silence the citation notice? == > Run this once: > parallel --citation > It takes less than 10 seconds to do and is thus comparable to an 'OK. Do not show this again'-dialog box seen in LibreOffice, Firefox and similar programs.

deleted 5y ago

[deleted]

WhatIsDukkha 5y ago

This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... It has a lot of features that can feel excessive at first glance but if you have felt some pain in building jobs, most of it is pretty sensible and much better than rollyourown.

cb321 5y ago
The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.
- ZoomZoomZoom 5y ago
  > Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Well, there was a Rust version with zero Perl, now unfortunately archived. It wasn't 100% on a par with the original and wasn't really finished. On the other hand, built easily for Windows and helped me on a few occasions. https://github.com/mmstick/parallel https://github.com/mmstick/parallel
  - cb321 5y ago
    Well, some archived project is not so great either. The core functionality is not even a 20 line bash script since bash grew wait -n, though: #!/bin/bash if [ "${1-0}" -lt 1 ]; then # No arg / arg not a number >= 1 echo "Usage: $0 <N>"; echo "reads cmds from stdin, running up to N at once." exit 1 fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1 while read cmd; do jobs > $TMP # jobs | wc -l does not work if [ $(wc -l < $TMP) -ge $1 ]; then wait -n # Wait for 1/more jobs to finish fi # Could accumulate total $? above, but would need to replace final wait. STRIPE_SEQ=$((STRIPE_SEQ + 1)) ( eval "$cmd" ) < /dev/null & # Run job in a subshell in bg done rm -f $TMP wait # Wait for all to finish and if bash ever grows some magic environment variable $NUM_BG_JOBS or you don't want auto-help or sequence numbers or etc. it can be even simpler.

dang 5y ago

Related: GNU Parallel 2018 - https://news.ycombinator.com/item?id=20726631 https://news.ycombinator.com/item?id=20726631 - Aug 2019 (68 comments) GNU Parallel Cheat Sheet [pdf] - https://news.ycombinator.com/item?id=19330356 https://news.ycombinator.com/item?id=19330356 - March 2019 (63 comments) GNU Parallel - https://news.ycombinator.com/item?id=13258142 https://news.ycombinator.com/item?id=13258142 - Dec 2016 (83 comments) GNU Parallel Tutorial - https://news.ycombinator.com/item?id=12943150 https://news.ycombinator.com/item?id=12943150 - Nov 2016 (65 comments) A Million Text Files and a Single Laptop - https://news.ycombinator.com/item?id=11248326 https://news.ycombinator.com/item?id=11248326 - March 2016 (27 comments) GNU Parallel – The command line power tool - https://news.ycombinator.com/item?id=6209767 https://news.ycombinator.com/item?id=6209767 - Aug 2013 (28 comments) GNU/Parallel changed my life - https://news.ycombinator.com/item?id=1894639 https://news.ycombinator.com/item?id=1894639 - Nov 2010 (8 comments) GNU Parallel - build and execute command lines from standard input in parallel - https://news.ycombinator.com/item?id=1801186 https://news.ycombinator.com/item?id=1801186 - Oct 2010 (36 comments)

arrakeen 5y ago

parallel is so useful and i use it multiple times daily. i wish its `:::` syntax was supported at the shell level so i could use it for every application

paskozdilar 5y ago

GNU Parallel is one of my favorite utilities of all time. I used to write complex and bug-ridden scripts that used bash job control to implement parallel execution of batch jobs. When I discovered GNU Parallel, I deleted them all and never looked back. Also, the documentation is awesome - there is a whole book [0] on GNU Parallel, and the manpage even links to a series of youtube videos [1] that explain how it works. [0] https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pdf https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pd... [1] http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

mekster 5y ago

If you want to execute commands on multiple remote hosts over SSH, just run a tmux session and launch a tab on each host in a loop and execute them. It's far easier to follow the output and individually deal with prompts.

permalac 5y ago
I do the same, I even have a command that tells me which tmux panel I'm in so I can ssh to many servers at once from tmux. However, when you just want to execute the same command : https://clustershell.readthedocs.io/en/latest/tools/clush.html https://clustershell.readthedocs.io/en/latest/tools/clush.ht...
paskozdilar 5y ago
True, but for a large number of hosts, going manually through prompts doesn't scale anymore. Sometimes you just want to check whether all hosts have the same configuration file, or something like that. That's where GNU Parallel can save your gluteus maximus: cat hosts.txt | parallel --quote --timeout=10 ssh {} 'echo {} $(md5sum ~/.config/file)' The above command would take a list of hostnames from the hosts.txt file, connect to each one, hash the config file and print out hostnames and hashes in one line per host.
- mekster 5y ago
  Unless entire operations can be finished automatically, you could still do, command_to_test && exit in the swarm of tmux tabs and let only the anomalies stay open.
- MainJane 5y ago
  cat hosts.txt | parallel --quote --timeout=10 ssh {} 'echo {} $(md5sum ~/.config/file)' Also try: parallel --slf hosts.txt --timeout=10 --nonall --tag md5sum .config/file
ktm5j 5y ago
I love tmux for this application, and typically I tend to prefer alternate solutions to using parallel.. but to be fair there are many more usecases for parallel than the one you describe here.

technofiend 5y ago

There's probably a more unix-centric / idiomatic way to do it, but using GNU Parallel along with Redis made automating tasks to process tens of thousands of systems extremely easy, repeatable, pausable, restartable, etc. And the fact you can do atomic operations on queues prevented any potential worker collisions.

winrid 5y ago

For those that don't know, it's written in Perl! Such a life saver of a tool. I used GNU Parallel to run a script locally that did what a big distributed system did, quicker and more reliably. It got to the point where people would just ask me to "run the thing" on my laptop instead of waiting for the cron. Mirror: https://github.com/martinda/gnu-parallel https://github.com/martinda/gnu-parallel

desktopninja 5y ago

Can't quite pin my resistance to parallel but I've enjoyed using this magic for many moons now: xargs -t -n 1 -P 3 -I TARGET_COMPUTER ssh -q TARGET_COMPUTER " sleep \$(shuf -i 1-9 | head -1) echo \$(hostname; date) echo \"===> \$(hostname) <===\" > ~/I_WAS_HERE " <<< " server-01 server-02 server-03 server-04 server-05 "

caro11ne 5y ago
I use: iwashere() { sleep $(shuf -i 1-9 | head -1) date echo "===> $(hostname) <===" > ~/I_WAS_HERE } env_parallel -Sserver-0{1..5} --tag --nonall iwashere I like that I can try out the function locally before running it remotely. I like that I do not have to give xargs a multiline argument. I am terrible at quoting inside quoting inside quoting, so I like that I can simply avoid the quoting. I also think my colleagues will find it easier to read (thus maintain).