8 ms·
Stop Piping Cats
- jrockway 17y agoThis is my least-favorite Internet meme. People "pipe cats" because they want the entire pipeline to read left-to-right, like: cat file | xargs foo | grep bar | sort | wc -l It just looks nicer than: < file xargs foo | grep bar ...
- gwern 17y agoI would also add that not all tools follow the '-' convention, and the ones that do can break in corner-cases. Why invest the effort of remembering how every tool works and which ones don't, for a obfuscating micro-optimization? /proud useless user of cat
- jrockway 17y agoExactly. I tried to figure out how to make xargs read from a file instead of stdin, and was unsuccessful. My "cat ... |" works every time (at the expense of 5 miliseconds of CPU time. oh noes.)
- camccann 17y agoThere's a nice consistency in doing it that way--it's very easy to think of cat as turning an "inert" file into a stream of data, each program as a function that applies a transformation to a stream with a particular kind of content, and the pipe as function composition (given compatible stream content). A very comfortable programming-like perspective, at least to me. (Pop quiz: What programming idiom closely resembles this "repackage content and apply sequential transformations" approach?)
- silentbicycle 17y agoSpoiler alert: See also Oleg Kiselyov's "Monadic i/o and UNIX shell programming" (http://okmij.org/ftp/Computation/monadic-shell.html http://okmij.org/ftp/Computation/monadic-shell.html).
- theblackbox 17y agoSeriously? You failed at that? I'm intrigued because it always occured to me that you were a particularly fastidious person (with all due respect). I often come accross issues that would seem for all the world to be a pointless artifact of an accepted convention. If this accepted convention is an acquired trait and one learns exclusivly from a limited meduim (irc frustrates me), I find it very curious that they persist as they sometimes act as a difficult barrier for a thorough understanding. This xargs thing, though I know almost nothing about it, seems to parallel similar problems I've encountered and I for one would be interested to hear your take on this (and HN in general). Or just some food for thought for your next blog post maybe?
- jrockway 17y agoWell, my guess is that xargs simply does not accept a filename as the argument. This comes from reading the --help and grepping the manpage for "file". The point was, when I used "cat file |", I knew it was going to work, and it did. When I tried to eliminate the cat by using the program's built-in ability to read a file, I had to read several manpages before determining that it was not possible. All because some tutorial says "cat is useless", when it clearly saved me much more time than the extra CPU time it used. And if you are actually asking; xargs is just a utility to read command-line-arguments from stdin. `echo -e "foo\nbar" | xargs rm' === `rm foo; rm bar' (or depending on the xargs implementation; `rm foo bar'). It kind of reminds me of a "functor map" operation, where stdin is a functor (of command-line arguments), and command-line programs are functions. (I will now mention that xargs also does "join" on the "results" of the "function", which is very ... monad-like. But "Monads are teh awesome and everything is one" is my second-least-favorite Internet meme, so I will spare you. :)
- olefoo 17y agoThe whole 'useless use of cat' meme is basically designed to let Randal Schwartz make fun of people. At one point it may have made a difference, and certainly if you have a shell script that is getting called 10,000 times a day it might make sense to optimise it. But for doing stuff from the commandline; weird shell acrobatics is a premature optimisation.
- Estragon 17y agoI tried to figure out how to make xargs read from a file instead of stdin, and was unsuccessful. "xargs -a <filename>" First option described in the man page on Red Hat.
- lg 17y agocould be a GNU vs. BSD thing.
- silentbicycle 17y agoBSD xargs doesn't seem to have the -a option.
- jrockway 17y agoMy home machine (Debian GNU/Linux) says "--arg-file", and it looks like a relatively new feature. My RHEL box at work definitely didn't have anything with the word "file" in it.
- gwern 17y agoThe whole thread descending from here is an excellent example of what I mean by corner-cases and memorization. :)
- spudlyo 17y agoSometimes you can use /dev/fd/0, but I agree, it's a stupid micro optimization, and I don't understand why people get so righteous about it. My co-worker would correct you every time he saw it, but then again he's a pedantic geek.
- pyre 17y agoIt's still good to keep it in mind, in case there are instances where it's being executed many times causing a bottleneck (though one-liners aren't really meant to be used in such situations).
- wendroid 17y agozsh and rc provide primitives for this, this is rc diff <{ps} <{sleep 3; ps}
- vidarh 17y agoEquivalent in bash: diff <(ps) <(sleep 3; ps)
- blasdel 17y agoSpeak for yourself, and I don't know anyone that puts the input redirection first. I do this: (xargs foo | grep bar | sort | wc -l) < file It makes the pipeline one command, both lexically (verb comes first) and concretely (the pipeline is kicked off in a subshell)
- barrkel 17y agoAt the expense of the data flowing from right to left to right, from the outside in. At least with cat, data flows unambiguously from left to right.
- blasdel 17y agoBut using cat totally fucks up the calling conventions: it puts the operand as the second of many arguments. Which one of these doesn't belong?: verb file cat file | verb1 | verb2 | verb3 (verb1 | verb2 | verb3) < file verb123 file EDIT: Consider the calling conventions, which one of these handles it's arguments differently? Assume that verb123 is an equivalent to the pipeline -- the subshell+stdin construction lends itself to shell aliases: alias verb123="(verb1 | verb2 | verb3) < "
- barrkel 17y agoI have no idea which one doesn't belong. Is 'verb123' the moral equivalent of '(verb1 | verb2 | verb3)'? In that case, it's the first one. But that's visible from the first word on the line, so I'm still not getting the picture. Of course, with the usual argument vs stdin conventions, either of the middle two could also be rewritten: verb1 file | verb2 | verb3 This is probably how I'd write such a chain.
- pyre 17y agoBut does the time it takes to launch the subshell equal the amount of time it takes to launch cat? If so your improvement is a NOOP.
- blasdel 17y ago
- hernan7 17y agoExactly. On a modern computer, the gain from not piping cats is almost always negligible. I don't think most shell scripts today ever get to the point where you have to worry about inner loops and the such. These days, if you hit a performance wall from spawning too many cats, I would think you switch to some scripting language where you have everything in one process. Premature optimization, people...
- dy9 17y agoWhat's the rationale for piping sort into wc?
- iigs 17y agoThat looks like it was a sample line that wasn't particularly useful, but if you do: cat file.txt | sort | uniq | wc -l ## notice uniq it gives you the count of unique lines. If you omit the sort in that instance it will fold duplicates together a/b/a is three lines, not two.
- gnosis 17y agoInstead of "sort | uniq" you could just use "sort -u"
- wendroid 17y agoThere's also the argument about debugging, one can insert a tee in any stage of cat file | xargs foo | grep bar | sort | wc -l
- aarongough 17y agoThe most useful thing I took away from this was not 'piping cats' but instead the interesting syntax for creating multiple directories in one go: ~ $ mkdir -p tmp/a/b/c ~ $ mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat/a}
- jerf 17y agoIt should be pointed out, since the article does not, that that is merely one example of shell expansion that can be used anywhere. It is not a mkdir feature. $ echo project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat/a} project/lib/ext project/bin project/src project/doc/html project/doc/info project/doc/pdf project/demo/stat/a (I added a linefeed to prevent wrapping.)
- aarongough 17y agoHuh, I'm definitely going to have to play with that! I only drop into a shell at most for 20 minutes a day at the moment, so a lot of the really neat time-saving trick simply don't stick in my head due to disuse...
- jerf 17y agoShells are one of those things that I find you should basically just plan on reading the man page every couple of months. Every time you do, you are virtually sure to discover something useful you'd swear you never saw before, even though you're pretty sure it's the exact same man page as before....
- tsuraan 17y agoAnother fun thing to do is use an empty entry, like mv .xinitrc{,.bak} or, if you have some stuff mounted under another mountpoint, such as when installing gentoo: umount /mnt/gentoo/{proc,dev,boot,} That empty entry expands to the base path, which can be a nice shortcut.
- silentbicycle 17y agoIt works in makefiles, too. (100% sure about BSD make, pretty sure about GNU make and others.)
- jerf 17y agoOh come on, you can't mention that without mentioning the Useless Use of Cat Award: http://partmaps.org/era/unix/award.html http://partmaps.org/era/unix/award.html
- Goladus 17y agoI use cat because often I wind up doing a number of different commands on the same file. It's a lot easier to edit the end of the line, especially if you're just adding another filter, than to go back and modify the beginning. eg cat file | less cat file | grep thing cat file | grep otherthing cat file | grep otherthing | cut stuff instead of less file grep thing file grep otherthing file grep otherthing file | cut stuff
- swolchok 17y agoSounds like you might want to learn about !$ (last token of previous command).
- Goladus 17y agoYeah that's interesting, but the second version is still more complicated to assemble and takes more keystrokes, assuming up arrow gives the entire previous command.
- swolchok 17y agoLet's count for your example. In all cases, I'll exclude the actual name of the file. We type in the first command: cat file | less less file The second version is 5 fewer keystrokes. On to the next command: cat file | grep thing grep thing !$ The second example is the same or fewer keystrokes. In both cases, you have to type "grep thing". In the first, you have to press the up arrow and backspace over "less" (at least three keystrokes), and in the second, you have to type an extra " !$". I'll skip moving to "cat file | grep otherthing" or "grep otherthing !$", and consider the change to get to cat file | grep otherthing | cut stuff grep otherthing file | cut stuff In both cases, you have to type " | cut stuff". If you key in the second example as "!! | cut stuff", that's an extra two keystrokes. If you key in the first as up arrow + "| cut stuff", that's only 1 extra keystroke. In total, my version saves keypresses in the specific example and doesn't seem much worse in general.
- prodigal_erik 17y ago
- tyrmored 17y agoGreat practical advice. I still pipe cats though because it's just more intuitive to me for setting up complex pipes.
- deleted 17y ago[deleted]
- chuhnk 17y agoThere is some really great info on that site. I'm a sys admin of 3 years and to see the comparison in performance is going to force me to change my scripting habits.
- mhansen 17y agoReally? You're changing your scripting habits based on gaining milliseconds?
- chuhnk 17y agoYou have to define usage here. They are giving an example of a single piped command on a file of who knows what size, probably not very big, hence milliseconds. However when it comes down to scripting large log processing, backups, and other forms of automation then I think it will yield better performance. Not to mention good coding practices, I am looking to improve in any form, something like this changes the way you think which will prove to be very useful when programming in ruby, c, java, etc. Not necessarily the use of piping commands, but just about how to be more efficient and less wasteful.
- pretz 17y agoThat 0.005 seconds I saved by not piping cat will significantly increase my productivity! No more wasting time!
- ynniv 17y agoYou also can't follow a file with grep, but you can with tail. Doing so requires the grep option "--line-buffered", which sacrifices some performance, but not much compared to actually viewing the log data. tail -f access.log | grep --line-buffered "GET /blog/post "
- randallsquared 17y agoI do this a lot, and I've never used "--line-buffered", nor apparently needed it. Google shows me a lot of people using that as a solution to a problem (not getting results immediately) I've never seen. Weird.
- ynniv 17y agoI only started using it because results were not appearing in "real-time", so it may depend on your OS, or your definition of "real-time".
- silentbicycle 17y agoIt's more likely if you have grep and several other operations piped together - the normal buffering leads to grep only printing every time it gets a full block of data, typically several lines.
- Zarkonnen 17y agohttp://everything2.com/title/I+am+forced+to+smoke+my+cat http://everything2.com/title/I+am+forced+to+smoke+my+cat - sorry!
- almost 17y agoWhy? The extra 3 letters of "cat" are usually going to take less time to type the mental overhead of "where do I put the input file for this command" or the additional pipeline reasoning. I actually sometimes do it one way and sometimes the other, whatever comes to my fingers first (which is going to vary according to how I'm visualizing the pipeline of tasks in my head). In general I far prefer the (it seems to me) more Unixy way of having lots of very simple commands and chaining them together over the use of extra options and arguments.