6 ms·

Parallel Shells with Xargs

61 points by oedmarap 6y ago

saagarjha 6y ago
You can actually do this with GNU Paral…oh, wait, I forgot that this isn't Stack Overflow and I'm not Ole. But seriously, xargs is pretty powerful by itself; I've seen a number of awkward pipelines using heavyweight tools when xargs would have sufficed.
- sidpatil 6y ago
  The article does mention GNU Parallel, but focuses on xargs.
philsnow 6y ago
I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator/operation kind of goes in the middle) and xargs is postfix (the op goes on the end). for file in *pdf *azw3; do calibre-convert $file $file.epub; done vs find . -name \*pdf -o -name \*azw3 -print0 | xargs -I {} calibre-convert {} {}.epub Something about the former is more natural for my brain to produce. If I want to parallelize, for shell loops my muscle memory very quickly converts the above to for file in *pdf *azw3; do calibre-convert $file $file.epub & done; wait ... but if I have a million files my machine is going to have a bad day, whereas the xargs version is easy to limit to N parallel invocations. I'd really like a shell that lets me define my own sugared versions of shell grammar. I imagine writing a 'forp' that would let me do forp $(nproc) file in ...; do ...; done and the shell would run up to $(nproc) invocations in parallel. Anybody know of such a shell? (Don't say eshell.)
- philsnow 6y ago
  forgot to mention that if the "operation" you're doing has any sort of conditions or branching, now you're writing ... | xargs -I{} bash -c "if [ $(stat --printf %Y {}) -gt 1234567890 ]; then do_something; else do_something_else; fi" whereas if you're building your command with just a shell loop, when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. I bet one could coerce emacs into doing something reasonable for syntax highlighting inside a quoted `xargs sh -c "..."` but ugh
  - JNRowe 6y ago
    Again, here we can rely on zsh having advanced globbing functionality. Instead of moving your test in to the execution block and suddenly having possibly unbalanced parallel groups you switch to something like *(.mw-1) to operate on files(the .) modified in the past week(the m[odified]w[eek]-1) or whatever. I'm choosing mtime because of your %Y, but zsh offers many other filters too. This is detailed in the glob qualifiers section of the zshexpn(1). I'm not sure whether I'd move the actual branching in because of the balancing issue, that is point where my hairs stand up. Perhaps, this is where it all falls apart.
  - cbsks 6y ago
    > when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. What?!? How have I never heard about this command? Thank you so much for telling me about it! This will be so much better than my current practice of manually copying unwieldy shell commands into a temporary file and running them there. https://en.wikipedia.org/wiki/Fc_(Unix) https://en.wikipedia.org/wiki/Fc_(Unix)
    - philsnow 6y ago
      One more fun thing in this vein: edit-and-execute-command is a readline binding commonly / by default bound to ^X^E (at least in "emacs mode"). It takes whatever command you're working on at the shell prompt and opens it with your $EDITOR (or $VISUAL if you have that defined separately for some reason). When you save and quit your editor, the shell runs the edited command.
  - waterhouse 6y ago
    And here we see an additional cost of the xargs approach: the $(stat ...) in the above will get expanded inside the ""s before it gets fed to xargs, and you would have to debug that (and ultimately either use single quotes or use \$). Nevertheless, I find xargs useful enough that I bite this bullet and have gotten very familiar with bash's quoting rules and how to work with them.
- JNRowe 6y ago
  This feels like exactly the reason I prefer using zsh's zargs command for these things. In part because you can use all the extended globbing functionality zsh provides. Something like the following [untested] rework of your pipeline: zargs -P $(nproc) -i ***(#i).(pdf|azw3) -- calibre-convert \{\} \{\}.epub We're firing up nproc tasks, matching case-insensitive .pdf or and .azw3 at any depth, and then batch processing them. zargs is documented in zshcontrib(1). It doesn't feel that different to your imagined syntax to me, quite whether that matches your mental model better is for you to decide though.
LukeShu 6y ago
The phrase "POSIX.2" is archaic. If POSIX is IEEE Std 1003, then "POSIX.1" is IEEE Std 1003.1, "POSIX.2" is IEEE Std 1003.2, and so on. In the first revisions, 1003.1 specified "Core Services" (mostly the C language stuff), and 1003.2 specified "Shell and Utilities". But when the Austin Group formed in 1997, they decided to roll the shell and utilities in to 1003.1. There no longer is a .2; it's been deprecated by 1003.1-2001 and all revisions since then.
- chasil 6y ago
  If POSIX.2 is archaic, then would you call POSIX job control precambrian? This reminds me of the stagnation of CMD.EXE, the origins of which appear to lie with OS/2.
  - LukeShu 6y ago
    Sure POSIX job control is old and janky, but it's still in modern POSIX.1, as it has been for 20 years now; while POSIX.2 has been dead for 20 years now.
    - chasil 6y ago
      The userland behavior of POSIX is separate from the C API. While I haven't read the latest standards, the only way to specifically address the userland is with "POSIX.2" (which is how the wiki does it). If there is a better way to say it, then the wiki should be updated.
      - chasil 6y ago
        Also, "The New Kornshell" (by Korn) has the following commentary in the "About this book" section: "ksh is intended to conform to the Shell Language Standard developed by the IEEE POSIX 1003.2 Shell and Utilities Language Committee." What term better describes the purview of that committee (alone) than POSIX.2?
        LukeShu 6y ago
gjvc 6y ago
More often than not, you'll get a better result (often more safely) by writing a small thing which generates the complete command lines for all the tasks which need running. Then once you are happy the right thing will be done, feed the list of commands to GNU parallel or something else which does the same thing. Not only does this approach provide confidence before pulling the trigger, but the list of commands may be retained forever for reference / example. It's also clearer and doesn't get the reader mystified by "shell wizardry", as some people are wont to be.
dhash 6y ago
Kind of embarrassing, but I never actually learned bash well enough to use it even after a decade of programming. When there was only one pair of feet in the sand, that’s where xargs carried me. Everything from web scrapers, to kubernetes admin, I’ve /only/ used xargs to do all of the variable splicing and parallelism that I need.

61 points by oedmarap 6y ago

You can actually do this with GNU Paral…oh, wait, I forgot that this isn't Stack Overflow and I'm not Ole. But seriously, xargs is pretty powerful by itself; I've seen a number of awkward pipelines using heavyweight tools when xargs would have sufficed.

sidpatil 6y ago
The article does mention GNU Parallel, but focuses on xargs.

philsnow 6y ago

I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator/operation kind of goes in the middle) and xargs is postfix (the op goes on the end). for file in *pdf *azw3; do calibre-convert $file $file.epub; done vs find . -name \*pdf -o -name \*azw3 -print0 | xargs -I {} calibre-convert {} {}.epub Something about the former is more natural for my brain to produce. If I want to parallelize, for shell loops my muscle memory very quickly converts the above to for file in *pdf *azw3; do calibre-convert $file $file.epub & done; wait ... but if I have a million files my machine is going to have a bad day, whereas the xargs version is easy to limit to N parallel invocations. I'd really like a shell that lets me define my own sugared versions of shell grammar. I imagine writing a 'forp' that would let me do forp $(nproc) file in ...; do ...; done and the shell would run up to $(nproc) invocations in parallel. Anybody know of such a shell? (Don't say eshell.)

philsnow 6y ago
forgot to mention that if the "operation" you're doing has any sort of conditions or branching, now you're writing ... | xargs -I{} bash -c "if [ $(stat --printf %Y {}) -gt 1234567890 ]; then do_something; else do_something_else; fi" whereas if you're building your command with just a shell loop, when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. I bet one could coerce emacs into doing something reasonable for syntax highlighting inside a quoted `xargs sh -c "..."` but ugh
- JNRowe 6y ago
  Again, here we can rely on zsh having advanced globbing functionality. Instead of moving your test in to the execution block and suddenly having possibly unbalanced parallel groups you switch to something like *(.mw-1) to operate on files(the .) modified in the past week(the m[odified]w[eek]-1) or whatever. I'm choosing mtime because of your %Y, but zsh offers many other filters too. This is detailed in the glob qualifiers section of the zshexpn(1). I'm not sure whether I'd move the actual branching in because of the balancing issue, that is point where my hairs stand up. Perhaps, this is where it all falls apart.
- cbsks 6y ago
  > when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. What?!? How have I never heard about this command? Thank you so much for telling me about it! This will be so much better than my current practice of manually copying unwieldy shell commands into a temporary file and running them there. https://en.wikipedia.org/wiki/Fc_(Unix) https://en.wikipedia.org/wiki/Fc_(Unix)
  - philsnow 6y ago
    One more fun thing in this vein: edit-and-execute-command is a readline binding commonly / by default bound to ^X^E (at least in "emacs mode"). It takes whatever command you're working on at the shell prompt and opens it with your $EDITOR (or $VISUAL if you have that defined separately for some reason). When you save and quit your editor, the shell runs the edited command.
- waterhouse 6y ago
  And here we see an additional cost of the xargs approach: the $(stat ...) in the above will get expanded inside the ""s before it gets fed to xargs, and you would have to debug that (and ultimately either use single quotes or use \$). Nevertheless, I find xargs useful enough that I bite this bullet and have gotten very familiar with bash's quoting rules and how to work with them.
JNRowe 6y ago
This feels like exactly the reason I prefer using zsh's zargs command for these things. In part because you can use all the extended globbing functionality zsh provides. Something like the following [untested] rework of your pipeline: zargs -P $(nproc) -i ***(#i).(pdf|azw3) -- calibre-convert \{\} \{\}.epub We're firing up nproc tasks, matching case-insensitive .pdf or and .azw3 at any depth, and then batch processing them. zargs is documented in zshcontrib(1). It doesn't feel that different to your imagined syntax to me, quite whether that matches your mental model better is for you to decide though.

LukeShu 6y ago

The phrase "POSIX.2" is archaic. If POSIX is IEEE Std 1003, then "POSIX.1" is IEEE Std 1003.1, "POSIX.2" is IEEE Std 1003.2, and so on. In the first revisions, 1003.1 specified "Core Services" (mostly the C language stuff), and 1003.2 specified "Shell and Utilities". But when the Austin Group formed in 1997, they decided to roll the shell and utilities in to 1003.1. There no longer is a .2; it's been deprecated by 1003.1-2001 and all revisions since then.

chasil 6y ago
If POSIX.2 is archaic, then would you call POSIX job control precambrian? This reminds me of the stagnation of CMD.EXE, the origins of which appear to lie with OS/2.
- LukeShu 6y ago
  Sure POSIX job control is old and janky, but it's still in modern POSIX.1, as it has been for 20 years now; while POSIX.2 has been dead for 20 years now.
  - chasil 6y ago
    The userland behavior of POSIX is separate from the C API. While I haven't read the latest standards, the only way to specifically address the userland is with "POSIX.2" (which is how the wiki does it). If there is a better way to say it, then the wiki should be updated.
    - chasil 6y ago
      Also, "The New Kornshell" (by Korn) has the following commentary in the "About this book" section: "ksh is intended to conform to the Shell Language Standard developed by the IEEE POSIX 1003.2 Shell and Utilities Language Committee." What term better describes the purview of that committee (alone) than POSIX.2?
      - LukeShu 6y ago

gjvc 6y ago

More often than not, you'll get a better result (often more safely) by writing a small thing which generates the complete command lines for all the tasks which need running. Then once you are happy the right thing will be done, feed the list of commands to GNU parallel or something else which does the same thing. Not only does this approach provide confidence before pulling the trigger, but the list of commands may be retained forever for reference / example. It's also clearer and doesn't get the reader mystified by "shell wizardry", as some people are wont to be.

dhash 6y ago

Kind of embarrassing, but I never actually learned bash well enough to use it even after a decade of programming. When there was only one pair of feet in the sand, that’s where xargs carried me. Everything from web scrapers, to kubernetes admin, I’ve /only/ used xargs to do all of the variable splicing and parallelism that I need.