3 ms·

I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator/operation kind of goes in the mi

I bristle when I have to switch from a shell for loop to piping jobs to xargs. For loops are like infix notation (the operator/operation kind of goes in the middle) and xargs is postfix (the op goes on the end). for file in *pdf *azw3; do calibre-convert $file $file.epub; done vs find . -name \*pdf -o -name \*azw3 -print0 | xargs -I {} calibre-convert {} {}.epub Something about the former is more natural for my brain to produce. If I want to parallelize, for shell loops my muscle memory very quickly converts the above to for file in *pdf *azw3; do calibre-convert $file $file.epub & done; wait ... but if I have a million files my machine is going to have a bad day, whereas the xargs version is easy to limit to N parallel invocations. I'd really like a shell that lets me define my own sugared versions of shell grammar. I imagine writing a 'forp' that would let me do forp $(nproc) file in ...; do ...; done and the shell would run up to $(nproc) invocations in parallel. Anybody know of such a shell? (Don't say eshell.)

philsnow 6y ago
forgot to mention that if the "operation" you're doing has any sort of conditions or branching, now you're writing ... | xargs -I{} bash -c "if [ $(stat --printf %Y {}) -gt 1234567890 ]; then do_something; else do_something_else; fi" whereas if you're building your command with just a shell loop, when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. I bet one could coerce emacs into doing something reasonable for syntax highlighting inside a quoted `xargs sh -c "..."` but ugh
- JNRowe 6y ago
  Again, here we can rely on zsh having advanced globbing functionality. Instead of moving your test in to the execution block and suddenly having possibly unbalanced parallel groups you switch to something like *(.mw-1) to operate on files(the .) modified in the past week(the m[odified]w[eek]-1) or whatever. I'm choosing mtime because of your %Y, but zsh offers many other filters too. This is detailed in the glob qualifiers section of the zshexpn(1). I'm not sure whether I'd move the actual branching in because of the balancing issue, that is point where my hairs stand up. Perhaps, this is where it all falls apart.
- cbsks 6y ago
  > when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. What?!? How have I never heard about this command? Thank you so much for telling me about it! This will be so much better than my current practice of manually copying unwieldy shell commands into a temporary file and running them there. https://en.wikipedia.org/wiki/Fc_(Unix) https://en.wikipedia.org/wiki/Fc_(Unix)
  - philsnow 6y ago
    One more fun thing in this vein: edit-and-execute-command is a readline binding commonly / by default bound to ^X^E (at least in "emacs mode"). It takes whatever command you're working on at the shell prompt and opens it with your $EDITOR (or $VISUAL if you have that defined separately for some reason). When you save and quit your editor, the shell runs the edited command.
- waterhouse 6y ago
  And here we see an additional cost of the xargs approach: the $(stat ...) in the above will get expanded inside the ""s before it gets fed to xargs, and you would have to debug that (and ultimately either use single quotes or use \$). Nevertheless, I find xargs useful enough that I bite this bullet and have gotten very familiar with bash's quoting rules and how to work with them.
JNRowe 6y ago
This feels like exactly the reason I prefer using zsh's zargs command for these things. In part because you can use all the extended globbing functionality zsh provides. Something like the following [untested] rework of your pipeline: zargs -P $(nproc) -i ***(#i).(pdf|azw3) -- calibre-convert \{\} \{\}.epub We're firing up nproc tasks, matching case-insensitive .pdf or and .azw3 at any depth, and then batch processing them. zargs is documented in zshcontrib(1). It doesn't feel that different to your imagined syntax to me, quite whether that matches your mental model better is for you to decide though.

philsnow 6y ago

forgot to mention that if the "operation" you're doing has any sort of conditions or branching, now you're writing ... | xargs -I{} bash -c "if [ $(stat --printf %Y {}) -gt 1234567890 ]; then do_something; else do_something_else; fi" whereas if you're building your command with just a shell loop, when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. I bet one could coerce emacs into doing something reasonable for syntax highlighting inside a quoted `xargs sh -c "..."` but ugh

JNRowe 6y ago
Again, here we can rely on zsh having advanced globbing functionality. Instead of moving your test in to the execution block and suddenly having possibly unbalanced parallel groups you switch to something like *(.mw-1) to operate on files(the .) modified in the past week(the m[odified]w[eek]-1) or whatever. I'm choosing mtime because of your %Y, but zsh offers many other filters too. This is detailed in the glob qualifiers section of the zshexpn(1). I'm not sure whether I'd move the actual branching in because of the balancing issue, that is point where my hairs stand up. Perhaps, this is where it all falls apart.
cbsks 6y ago
> when things start to get complicated you can `fc` into your $EDITOR, with pretty-print (with newlines) and syntax highlighting or whatever else you want. What?!? How have I never heard about this command? Thank you so much for telling me about it! This will be so much better than my current practice of manually copying unwieldy shell commands into a temporary file and running them there. https://en.wikipedia.org/wiki/Fc_(Unix) https://en.wikipedia.org/wiki/Fc_(Unix)
- philsnow 6y ago
  One more fun thing in this vein: edit-and-execute-command is a readline binding commonly / by default bound to ^X^E (at least in "emacs mode"). It takes whatever command you're working on at the shell prompt and opens it with your $EDITOR (or $VISUAL if you have that defined separately for some reason). When you save and quit your editor, the shell runs the edited command.
waterhouse 6y ago
And here we see an additional cost of the xargs approach: the $(stat ...) in the above will get expanded inside the ""s before it gets fed to xargs, and you would have to debug that (and ultimately either use single quotes or use \$). Nevertheless, I find xargs useful enough that I bite this bullet and have gotten very familiar with bash's quoting rules and how to work with them.

JNRowe 6y ago

This feels like exactly the reason I prefer using zsh's zargs command for these things. In part because you can use all the extended globbing functionality zsh provides. Something like the following [untested] rework of your pipeline: zargs -P $(nproc) -i ***(#i).(pdf|azw3) -- calibre-convert \{\} \{\}.epub We're firing up nproc tasks, matching case-insensitive .pdf or and .azw3 at any depth, and then batch processing them. zargs is documented in zshcontrib(1). It doesn't feel that different to your imagined syntax to me, quite whether that matches your mental model better is for you to decide though.