4 ms·

A small primer on xargs

48 points by bitops 14y ago

deleted 14y ago
[deleted]
DHowett 14y ago
"In particular, the presence of \0 in your data will throw xargs for a loop (no pun intended). In this case, it is necessary to use the -0 option. This is particularly relevant if you are using the find command and supplying the -print0 option." This glosses over the fact that those two things are intended to be used together. Delimiting filenames with a \0 is superior to delimiting them with spaces or newlines, as filenames cannot usually contain ASCII NUL; xargs and find can therefore be used to remove heterogeneous lists of files. Additionally, the example given may better be worded as "find . -name '* junk* -exec rm '{}' \;" or even "find . -name '* junk* | xargs rm". (as given it strikes me as a useless use of grep and is additionally fragile if ls is aliased to "ls --color=always", spewing ANSI into EDIT: Please excuse the additional spaces in the find command arguments, they are a formatting avoidance feature.
- kjetil 14y ago
  Little known fact: the -0/-print0 idea is due to DJB.
- darklajid 14y ago
  If we're ripping this sample apart, I'd like to point out that rm *junk* in this case would've done the same, without spawning lots of useless stuff. xargs is cool. The example fails to show a decent use case. For me a line like that would indicate that someone is _not_ familiar with his tools. Similar to cat somefile | grep something instead of grep something somefile Pipes and processes aren't endangered or (in practice) limited, but we still don't need to be wasteful?
  - babarock 14y ago
    Furthermore, parsing the output of `ls` is a terrible practice[1] that makes for an even poorer sample. A little further in the article, the author even suggests running `ls | grep junk` (instead of the much simpler `ls junk*`. Do I need to point that out?). I also know some people are going to retort that wasting a process or a pipe is not that bad. This discussion has been had over and over again, probably ever since Unix was invented. In any case, the poor example chosen, as well as the flaky commands used, seriously hinder the credibility of the article. [1]: http://mywiki.wooledge.org/ParsingLs http://mywiki.wooledge.org/ParsingLs
    - emmelaich 14y ago
      They don't do the quite the same thing. ls | grep junk is more similar to ls -d *junk* You can argue the first is simpler. You can also argue more successfully that it is more general. edit: I acknowledge the points in your link. Advocates for the unix "bunch of small tools" philosophy tend to overlook its weak spots.
- lubomir 14y ago
  Instead of -exec rm {} \; you could use -delete option as well.
  - babarock 14y ago
    This option is not portable at all. For instance, it won't work on Solaris' find. I strongly advise against the use of non portable options.
    - exDM69 14y ago
      It may be a good idea when writing scripts that have to be portable. But there's quite a lot of nice non-standard options in the GNU userspace tools, and it would be a shame not to use them when available.
- asdfaoeu 14y ago
  If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces. Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.
  - deleted 14y ago
    [deleted]
  - robertskmiles 14y ago
    > more likely is its aliased to "ls --color=auto" or similar which will work normally. My ls (which I haven't much reconfigured) behaves very differently depending on whether it's outputting to a shell or a pipe or redirect. Not just colours, but columning and filetype indicators as well.
djoe48 14y ago
ls | grep junk | xargs rm # is dangerous due to The separator problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem.
- rcthompson 14y ago
  ls | grep junk | xargs trash-put https://github.com/andreafrancia/trash-cli/#readme https://github.com/andreafrancia/trash-cli/#readme
webreac 14y ago
I try to use perl instead of sed, awk, bash each time the needs are beyond basic usage. This applies also sometimes for xargs for example with the following command. find . -name junk\* | perl -nle unlink My favorite usage of xargs if for parallel operations: cat hostlist | xargs -P 0 -n 1 -I host rsh host gzip -9 log/*.log
slashclee 14y ago
xargs is so 20th-century. parallel[0] is way more awesome (especially with all the multi-core systems everybody has now). Cool bonus: all the things you learn for xargs work in parallel too! [0]: http://www.gnu.org/software/parallel/ http://www.gnu.org/software/parallel/
- wwwhizz 14y ago
  I Was wondering what the difference with running parallel jobs using xargs -P was, but that's covered in their man-page: http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel http://www.gnu.org/software/parallel/man.html#differences_be...
- knewter 14y ago
  A billion times, this. I just learned of GNU Parallel about 3 months ago and wish everyone knew about it. Have seen it used to stunning effect with a comically large pipeline (OCR patents) across a comically large number of EC2 instances. Very awesome indeed.
gizzlon 14y ago
"What xargs is doing is taking each line of input and applying the supplied argument or command to the line of input" Actually it does not run command once for each line of input. This confused me for some time until I read the man page: "The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option" Think you're looking for "-n 1" if you want that behavior..
- leed25d 14y ago
  I was going to respond on this very point myself. This blog entry is just plain wrong wrt the way that xargs works.
bitops 14y ago
OP here - thanks all for the comments - so yes, it looks like I don't know xargs as well as I thought I did. I'll update the post to be more accurate. As for the points about a bad example, I am fully aware that it is extremely contrived. The point was to not confuse anyone with commands they don't already know. I'm wary of posting examples that necessitate people learning 4-5 new commands when their focus is learning just one.

48 points by bitops 14y ago

deleted 14y ago

[deleted]

"In particular, the presence of \0 in your data will throw xargs for a loop (no pun intended). In this case, it is necessary to use the -0 option. This is particularly relevant if you are using the find command and supplying the -print0 option." This glosses over the fact that those two things are intended to be used together. Delimiting filenames with a \0 is superior to delimiting them with spaces or newlines, as filenames cannot usually contain ASCII NUL; xargs and find can therefore be used to remove heterogeneous lists of files. Additionally, the example given may better be worded as "find . -name '* junk* -exec rm '{}' \;" or even "find . -name '* junk* | xargs rm". (as given it strikes me as a useless use of grep and is additionally fragile if ls is aliased to "ls --color=always", spewing ANSI into EDIT: Please excuse the additional spaces in the find command arguments, they are a formatting avoidance feature.

kjetil 14y ago
Little known fact: the -0/-print0 idea is due to DJB.
darklajid 14y ago
If we're ripping this sample apart, I'd like to point out that rm *junk* in this case would've done the same, without spawning lots of useless stuff. xargs is cool. The example fails to show a decent use case. For me a line like that would indicate that someone is _not_ familiar with his tools. Similar to cat somefile | grep something instead of grep something somefile Pipes and processes aren't endangered or (in practice) limited, but we still don't need to be wasteful?
- babarock 14y ago
  Furthermore, parsing the output of `ls` is a terrible practice[1] that makes for an even poorer sample. A little further in the article, the author even suggests running `ls | grep junk` (instead of the much simpler `ls junk*`. Do I need to point that out?). I also know some people are going to retort that wasting a process or a pipe is not that bad. This discussion has been had over and over again, probably ever since Unix was invented. In any case, the poor example chosen, as well as the flaky commands used, seriously hinder the credibility of the article. [1]: http://mywiki.wooledge.org/ParsingLs http://mywiki.wooledge.org/ParsingLs
  - emmelaich 14y ago
    They don't do the quite the same thing. ls | grep junk is more similar to ls -d *junk* You can argue the first is simpler. You can also argue more successfully that it is more general. edit: I acknowledge the points in your link. Advocates for the unix "bunch of small tools" philosophy tend to overlook its weak spots.
lubomir 14y ago
Instead of -exec rm {} \; you could use -delete option as well.
- babarock 14y ago
  This option is not portable at all. For instance, it won't work on Solaris' find. I strongly advise against the use of non portable options.
  - exDM69 14y ago
    It may be a good idea when writing scripts that have to be portable. But there's quite a lot of nice non-standard options in the GNU userspace tools, and it would be a shame not to use them when available.
asdfaoeu 14y ago
If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces. Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.
- deleted 14y ago
  [deleted]
- robertskmiles 14y ago
  > more likely is its aliased to "ls --color=auto" or similar which will work normally. My ls (which I haven't much reconfigured) behaves very differently depending on whether it's outputting to a shell or a pipe or redirect. Not just colours, but columning and filetype indicators as well.

djoe48 14y ago

ls | grep junk | xargs rm # is dangerous due to The separator problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem.

rcthompson 14y ago
ls | grep junk | xargs trash-put https://github.com/andreafrancia/trash-cli/#readme https://github.com/andreafrancia/trash-cli/#readme

webreac 14y ago

I try to use perl instead of sed, awk, bash each time the needs are beyond basic usage. This applies also sometimes for xargs for example with the following command. find . -name junk\* | perl -nle unlink My favorite usage of xargs if for parallel operations: cat hostlist | xargs -P 0 -n 1 -I host rsh host gzip -9 log/*.log

slashclee 14y ago

xargs is so 20th-century. parallel[0] is way more awesome (especially with all the multi-core systems everybody has now). Cool bonus: all the things you learn for xargs work in parallel too! [0]: http://www.gnu.org/software/parallel/ http://www.gnu.org/software/parallel/

wwwhizz 14y ago
I Was wondering what the difference with running parallel jobs using xargs -P was, but that's covered in their man-page: http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel http://www.gnu.org/software/parallel/man.html#differences_be...
knewter 14y ago
A billion times, this. I just learned of GNU Parallel about 3 months ago and wish everyone knew about it. Have seen it used to stunning effect with a comically large pipeline (OCR patents) across a comically large number of EC2 instances. Very awesome indeed.

gizzlon 14y ago

"What xargs is doing is taking each line of input and applying the supplied argument or command to the line of input" Actually it does not run command once for each line of input. This confused me for some time until I read the man page: "The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option" Think you're looking for "-n 1" if you want that behavior..

leed25d 14y ago
I was going to respond on this very point myself. This blog entry is just plain wrong wrt the way that xargs works.

bitops 14y ago

OP here - thanks all for the comments - so yes, it looks like I don't know xargs as well as I thought I did. I'll update the post to be more accurate. As for the points about a bad example, I am fully aware that it is extremely contrived. The point was to not confuse anyone with commands they don't already know. I'm wary of posting examples that necessitate people learning 4-5 new commands when their focus is learning just one.