6 ms·
I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewe
by fiddlerwoaroof 5y ago
I frequently find myself reaching for this pattern instead of xargs:
do_something | ( while read -r v; do
. . .
done )
I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)
- tomcam 5y agoThank you. Your comment coalesced a number of things in my mind that I hadn’t grasped properly as a UNIX midwit, especially the braces thing.
- ptspts 5y agoFor thousands of arguments this sloution is much slower (high CPU usage) than xargs, because either it implements the logic as a shell script (slow) or it runs an external program for each argument (slow).
- fiddlerwoaroof 5y agoSure, if performance matters use xargs. I find this is easier to read and think about.
- aaaaaaaaaaab 5y agoSome additional tips: 1. You don't need the parentheses. 2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope: i=0 while read -r v; do ... i=$(( i + 1)) done < <(do_something) The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯ [1] https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html https://www.gnu.org/software/bash/manual/html_node/Process-S...
- fiddlerwoaroof 5y agoYeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too. I think the redirection can come first, though (not at a computer to test): < <( do_something ) while read . . .
- aaaaaaaaaaab 5y agoYeah, for commands, the input/output redirections can precede them, but for some reason it doesn't work for builtin constructs like `while`: $ < <( echo foo ) while read -r f; do echo "$f"; done -bash: syntax error near unexpected token `do' $ < <( echo foo ) xargs echo foo $ bash --version GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin20.2.0)
- fiddlerwoaroof 5y agoMaybe wrap the loop either with parentheses or braces?
- aaaaaaaaaaab 5y agoTried that, but nope :D I'll let you figure this one out once you get near a computer!
- entire-name 5y agoRedirection like this doesn't seem to work if it comes first on GNU bash 5.0.17(1)-release. For documentation purposes, this is the exact thing I tried to run: $ < <(echo hi) while read a; do echo "got $a"; done -bash: syntax error near unexpected token `do' $ while read a; do echo "got $a"; done < <(echo hi) got hi Maybe there is another way...
- JNRowe 5y agoOne way which isn't great, but an option nonetheless… The zsh parser is happy with that form: $ zsh -c '< <(echo hi) while read a; do echo "got $a"; done' got hi My position isn't that it is a good reason to switch shells, but if you're using it anyway then it is an option.
- fiddlerwoaroof 5y agoI’ve always preferred zsh and, as I’ve slowly adopted nix, I’ve slowly stopped writing bash in favor of zsh
- chriswarbo 5y agoI use this exact pattern a lot. One thing to consider is that in the process substitution version, do_something can't modify the enclosing variables. The vast majority of the time I want to modify variables in the loop body and not the generating process, but it's worth keeping in mind. One common pattern I use this for is running a bunch of checks/tests, e.g. EXIT_CODE=0 while read -r F do do_check "$F" || EXIT_CODE=1 done < <(find ./tests -type f) exit "$EXIT_CODE" This is a more complicated alternative to the following: find ./tests -type f | while read -r F do do_check "$F" || exit 1 done The simpler version will abort on the first error, whilst the first version will always run all of the checks (exiting with an error afterwards, if any of them failed)
- fiddlerwoaroof 5y agoI usually write zsh scripts and I think there’s a shell option in zsh that allows the loop at the end of the pipe to modify variables in the enclosing body: I remember at least one occasion where I was surprised about this discrepancy between shells.
- aaaaaaaaaaab 5y agoInteresting! Indeed, Greg's BashFAQ notes it too: https://mywiki.wooledge.org/BashFAQ/024 https://mywiki.wooledge.org/BashFAQ/024 >Different shells exhibit different behaviors in this situation: >- BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>'). >- BASH, Yash and PDKsh-derived shells create a new process only if the loop is part of a pipeline. >- KornShell and Zsh creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. The read example above actually works in ksh88, ksh93, zsh! (but not MKsh or other PDKsh-derived shells) >- POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).
- vbezhenar 5y ago
- aaaaaaaaaaab 5y agoAlso for the `while` enthusiasts, here's how you zip the output of two processes in bash: paste -d \\n <(do_something1) <(do_something2) | while read -r var1 && read -r var2; do ... # var1 comes from do_something1, var2 comes from do_something2 done
- thayne 5y agocreating a subshell can lead to some surprising behavior if you aren't careful though.