8 ms·
Things you (probably) didn't know about xargs
- z_ 15y agoYou can skip the pipe by using find's '-exec' option. It will save you the pipe. The example: "find . -name '* .py' | xargs grep 'import'" would become "find . -name '* .py' -exec grep -H 'import' {} \;". You need to include the -H in grep to get the filename in which the match occurs.
- bellaire 15y agoUsing xargs with a pipe is easier. I don't know any reason why I'd want to "save a pipe" when working at the command line. Also note that text surrounded by asterisks in your comment become italics. Indent text by two or more spaces to reproduce text verbatim, like for code.
- ibejoeb 15y agoI think it was more about the appropriateness of the examples. z_'s comment is right-on, and I was going to post the same thing. It's a good intro, but the examples are contrived because you just don't need it. It's not necessarily about saving a pipe, but also, when the tool provide first-class support for the function, it's typically less prone to error. For example, the -print0 becomes unnecessary, and I've been burnt by that. I also appreciate the writeups that don't teach poor examples. We all know how prolific copy&paste coding is. How many times have you seen "grep foo bar | wc -l" when you know it's just all-around better to "grep -c foo bar"?
- cstejerean 15y agoI felt bad using contrived examples, but I wanted a short post to cover the basics of xargs without getting detracted with discussions of the options of find. Based on the feedback I've seen here I'll go ahead and update the post though. If you can share some simple but less contrived examples please let me know, I'd love to update the post.
- ibejoeb 15y agoDon't get me wrong; it's a perfectly good and useful tutorial. The meat comes at the end when you talk about parallelism and argument batching. That can make a world of difference when you're working on real-world problems, like moving millions of files (mv * won't work unless you're on a system without ARG_MAX, and even then there are performance implications). I think a good intro to xargs starts with a list of things that you can't do without it. (Easy for me to say that, but of course I haven't written that piece...) It'd be great to know why to use it, not just how, you know what I mean? Anyway, this is just off-the-cuff commentary, not criticism. Thanks for writing it up.
- prodigal_erik 15y agoI would prefer we only teach "grep -c" as a special case optimization to people who already understand how "grep | wc -l" works, because the latter is more generally useful.
- kaens 15y agoI guess the question then becomes "why is xargs with a pipe easier?", or "why do you view xargs with a pipe as easier?".
- DanielRibeiro 15y agoAck-grep[1] is also really nice for making such find/grep operations a simple command. [1] http://betterthangrep.com/ http://betterthangrep.com/
- skimbrel 15y agoAck is awesome. It makes life as a programmer so much easier -- no more fiddling around trying to remember the useful flags for grep. Plus it has --thpppt.
- ptramo 15y agoPlease stop calling it ack-grep. Debian already had an ack package and didn't get their priorities the way I would have wished. That's their problem, not the official name.
- dfc 15y ago"didn't get their priorities the way I would have wished"??? They already had a package named ack. WTF? By "getting their priorities" correct you mean catering to you? The same problem is currently happening with node.js. Its unfortunate that node.js chose node to replace an equally ambiguous and generic name.
- ptramo 15y agoI didn't say "correct". I said "the way I would have wished". I expressed a personal preference, you're making me sound like more of a jerk than I already am. That being said, yes, I believe they're plain wrong on this one. They shouldn't cater to me, but to their users. Thank God they have stats. http://qa.debian.org/popcon.php?package=ack-grep http://qa.debian.org/popcon.php?package=ack-grep http://qa.debian.org/popcon.php?package=ack http://qa.debian.org/popcon.php?package=ack I count in both because I installed the ack package mistakingly. I'm probably not the only one. Still a very large WIN for ack, the one I care about.
- Jach 15y agoOne of the reasons I love xargs so much is the archaic syntax (and surprises based on your input) behind -exec. I think the pipe is cleaner, or using find in a set of backticks.
- rubinelli 15y ago"find . -name '* .py' -exec grep 'import' {} +" will work more like the xargs example, i.e. all filenames will be passed as parameters to a single grep process, so the filename is displayed by default. I usually don't bother with this kind of command line micro-optimization, but in my experience find and grep are the commands for which it's worth knowing and using every option.
- albemuth 15y agoNot sure if this is a zsh thing only, but I do this: grep import **/*.py
- sjs 15y agoI rarely use find because zsh has great globbing. I really wish zsh were the default shell everywhere, most people wouldn't even notice if you changed the prompt from % to $
- e40 15y agoIt's more efficient to use the pipe with xargs. If you're removing 10,000 files, the difference in time between -exec (and fork/exec of rm 10,000 times) and xargs rm is quite significant. (On my system getconf MAX_ARGS is 2180000.)
- vrotaru 15y agoYou save one pipe, but at expense of every grep will spawn its own process. So maybe a 1000 grep processes instead of one. But I've saved a pipe.
- ams6110 15y agoDeveloping some proficiency with commands such as find, xargs, sed, grep, etc. is where you can really gain productivity. There are studies showing that the perception that the command line is more efficient than a GUI is false; the GUI is actually faster. This may be true for relatively simple operations such as moving, copying, or deleting files, but by involving find, xargs, and a few pipes you can quickly accomplish operations that would be either very tedious or flatly impossible in most GUI file "explorers".
- rjbond3rd 15y ago> There are studies showing that...the GUI is actually faster Can you elaborate? I'm trying to understand how studies can show the GUI is faster than the command line, with or without xargs, grep, etc.
- kaens 15y agoEvery study I've seen like this is indicating that they're faster to learn for the average joe, and possibly faster for certain types of tasks (stretching an image), not that they're faster in general -- I'm not even sure what exactly that would mean.
- nodata 15y ago> There are studies showing... Sources for the studies?
- botker 15y agoNever pipe `find` to `xargs rm -f`. All you need is a malicious empty file called / and you're screwed. That's why `find` has a -delete flag.
- tszming 15y agoA forward slash is not a valid file name.
- Jach 15y agoThe chances of such a file coming into existence accidentally and also being picked up in the find filter seem slim. If it's around maliciously, I think you have bigger problems. Anyway, there are many other failure modes of rm -rf that are a lot more likely and you should worry about.
- moe 15y agoThe chances of such a file coming into existence accidentally When sharing servers with less UNIX savvy developers you will observe A) a notoriously polluted home-directory and B) plenty of files with funny names such as -, * , user@server.com (from failed scp attempts) all over the place. It's indeed relatively hard to create a file called '/' by accident. But I've seen files containing the '/' along with spaces, which can be just as deadly. Oh and the popular * -file is not to be taken lightly either. As a rule of thumb: Learn the safe way to chain these commands (find/xargs in particular) once and stick to it. And always perform a dry-run before launching the real deal.
- koenigdavidmj 15y agoUnix filenames can not include / characters or null bytes, so I call shenanigans. Nonetheless your point stands: be careful. (And make liberal use of -print0 !)
- moe 15y agoSeems you are right! I could've sworn I've seen them before, guess my memory got mixed up there.
- tszming 15y agoUse GNU Parallel if you need to execute jobs in parallel (http://www.gnu.org/software/parallel/ http://www.gnu.org/software/parallel/)
- sabat 15y agoRecursively find all Emacs backup files and remove them find . -name '~' | xargs rm* Recursively find all Python files and search them for the word ‘import’ find . -name '.py' | xargs grep 'import'* Hmm. I don't mean to be a tweak, but you don't need xargs to do either of those things. Just: find . -name '~' -delete find . -name '.py' | grep 'import' Note: I can't figure out how to get an asterisk to show up, and don't have time to look it up.
- ptramo 15y agoNo. You're looking for files having import in their path here, not in their contents.
- sjs 15y agoHe got the command wrong but his point stands: find . -name '.py' -exec grep -H import '{}' \;
- Jach 15y agoIn addition to ptramo, you really want: grep 'import' `find -name '* .py'`
- ptramo 15y agoWell, you want xargs as soon as your project is big, because you'll reach the limits on arguments fairly quickly. I won't go too deep on the issues this will trigger if your filenames contain special characters. We often think of spaces, but you could have some nastier stuff. If you want to sound smart at your next geeks reunion, simply read http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h...
- ptramo 15y agoA few issues with this article. No, GNU xargs does not truncate the generated command. It will span multiple commands. Nor is it "-print 0" but "-print0" in find. And as mentioned in other comments, GNU parallel is much better for job parallelization.
- cstejerean 15y agoYou are right, I mistakenly assumed xargs truncated the command because the only thing I saw on the screen was the output of the last invocation. GNU parallel is great but xargs is installed by default on OS X, BSD. I'll go ahead and update the post and fix the -print0.
- uggedal 15y agoOne thing that have bitten me in the past is the fact that xargs always runs at least once, even if there are no input. By using the -r / --no-run-if-empty flag, xargs does not run the command if the input does not contain any nonblanks.
- ptramo 15y agoAs with many cool features, it's a GNU-only option. One can refer to http://pubs.opengroup.org/onlinepubs/009695399/utilities/xargs.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/xar... for the portable flags.
- gcb 15y agofor a long time I avoid xargs like the plague. much less error prone to user a bash loop or something.... but that parallelization parameter may win me back as it's cleaner than & and global vars for counters....
- gte910h 15y agoThe trace param to Xargs is nice too: You can check what will happen before doing something.
- RexRollman 15y agoThe best thing about Unix, and maybe the worst, is that you never stop learning about it. Nice article.
- billpg 15y agoI wonder why -print0 / -0 isn't the default, as it seems that not using those options is the wrong way to do it. (Either that or why filenames can have spaces or LFs in them.)
- ams6110 15y agoHistorically, UNIX filenames almost never had spaces in them.
- barrkel 15y agoIt's not the default because things like 'find | grep foo' wouldn't work as well: the output would appear all concatenated, as the terminal doesn't break lines on null characters. I wrote a utility I named print0, which simply converts line-oriented input into null-terminated output. Very useful for building pipelines of line-oriented utilities, where each line is a filename. It's quite common to have spaces in filenames (if nothing else, user files on NAS shares), and vanishingly rare to see newlines, so I find it to be a sensible tradeoff. Things like 'find | egrep | sed | print0 | xargs -0' work as you'd expect.
- premchai21 15y agoWould not « print0() { tr '\n' '\0'; } » work in most cases? Does your utility do something special to existing instances of \0? That's the only case I can imagine where the transformation is necessarily lossy and it's not clear what to do. I suppose in theory you could have newline normalization or something too.
- barrkel 15y agoNo, because that doesn't handle command-line arguments or '-' in the same way that cat does. It also doesn't handle the differences between DOS, Unix and Mac (\r, \n and \r\n) line endings properly. Finally, it's also slower than my 80-line C version.
- billpg 15y agoIf filenames can contain LF characters, 'find | grep foo' is also wrong. (Interesting idea with print0, but I fear this is mere symptomatic relief, rather than actually fixing the problem.)
- onedognight 15y agoWhile not the point of the article, I was happy to learn about bash's {..} operator! I've always missed perl's .. in the shell. echo {1..100}
- st3fan 15y agoNote that the first three examples can also be written in a much shorter form if you use zsh: wc -l **/*.py rm **/*~ grep 'import' **/*.py I like zsh a lot for this small feature.
- scrame 15y agoI like zsh for the same reason, but you will want to quote or escape the tilde in your second example. rm **/*\~ rm '**/*~'
- tlrobinson 15y agoI was just trying to figure out how to change the position of the arguments yesterday. I figured it out (thanks, man), but discovered the OS X version (FreeBSD?) uses -I while certain GNU versions use -i. Lame.
- james2vegas 15y agoIn GNU utils, incompatible features and extensions are a feature, not a bug.
- gnosis 15y agozsh and its zargs command can easily overcome many of the limitations of xargs noted in the article. They also make the use of the "find" command unnecessary.
- underwater 15y agoOn a slightly related note, am I the only one who finds the argument syntax for `find` to be inconsistent? Shouldn't it be `find --name "*.bar"` or even `find --name=foo.bar`?
- zaphar 15y agofinds arguments might very well be inconsistent but the --name argument as you show it does exist. -iname just means find should treat the name as case insensitive. --name on the other hand is case sensitive so --name=Foo.bar won't match foo.bar.
- kaens 15y agoThe "single - for single-letter -- for fullname" option style was popularized by gnu getopt, by far the most popular style. Find has been around for a while and probably doesn't update to that style because of the huge amount of stuff already using find. It's inconsistent in comparison to most utilities, for sure. It's consistent internally, and POSIX-compliant, at least. (iirc)