7 ms·
Useless Use of Cat Award
- joshbaptiste 13y agoYep, I see too many senior people violate UUOC/UUOG ie.. cat file | grep bar | awk '{ print $1 }' | sort | uniq which can be awk '/bar/ { print $1 | "sort -u"}' file The best resource if you really want to be a better shell citizen (assuming you use Bash) overall is #bash on irc.freenode.net and Greg's bash wiki. http://mywiki.wooledge.org/BashFAQ http://mywiki.wooledge.org/BashFAQ Honorable mention also for #awk on irc.freenode.net also which has taught me a lot.
- clicks 13y agoI would still use the former. Chances are, I'm not going to remember awk's somewhat weird, more unfamiliar syntax -- everyone knows about cat, everyone knows about piping, those two go very well together. Using cat with pipes typically linearizes $x operation in our minds, it makes it easier to come up with solutions, and makes it easier to change them as modification is needed.
- maw 13y agoUsing cat with pipes typically linearizes $x operation in our minds, it makes it easier to come up with solutions, and makes it easier to change them as modification is needed. Yes, definitely. Also it's easier to build up from simple to more complicated pipelines that way, testing whether each iteration does what you expect it to.
- sliverstorm 13y agoI don't think it's fair to get awk involved. It's a scripting language, not a single-purpose utility.
- e12e 13y agoIt's silly when set up like that. But I still prefer the separate invocations -- they become building blocks. It means I can replace the "cat file" with "echo testdata", it reads nice, and for more complicated scripts, it's easy to format across lines: zcat /var/log/somdeamon/* \ | grep "simple event like maybe 'error'" | grep "complicated date expression or whatever" | some other filter | summation? | maybe sort and uniq I can start that with a simple (z)cat of one file (rather than the thousands of lines I might want to work on) -- and it doesn't require much arcane knowledge of each of the tools used, for each step. I do agree that awk might be a little under-appreciated, but if you really feel that way, maybe you should just use snobol. To me a "wasted cat" clearly states intention: get/feed data. Then the rest of the steps either filter, sum, or manipulate data. The final thing that makes the "waste"-argument silly, is that Linux[1] has pretty lightweight processes. I'm not sure what the state of the various BSDs are, but it'd surprise if at least FreeBSD hasn't gotten a lot better on this too. Either way, one would think that any text pipeline would be io-bound, not limited by the number of forks... [1] http://bulk.fefe.de/scalability/ http://bulk.fefe.de/scalability/
- jl6 13y agoYour first line of code is much better. It optimizes for clarity, presenting the programmer's intention in a simple logical sequence. It also likely optimizes for programmer time and mental expenditure.
- abjr 13y agoWhy not: awk '/bar/ { print $1 }' file | sort -u
- laumars 13y agoI find those kind of examples a little unfair as what you're actually doing is comparing the syntax of different shells/languages (the Bourne Shell vs awk). You could write examples in Perl or Python as well - to prove how pointless awk and sort is as well if you really wanted. In fact you could even rewrite sort in awk. While I disagree with most of the comments on here about how: cat $file | grep "pattern" is easier to read than grep "pattern" $file (in my opinion, if you're a sysadmin then you should know how grep works and thus the optimised solution shouldn't be any harder to read. But at the end of the day, whatever gets the job done). However I do draw the line at your example because you're now comparing apples to oranges and suggesting that everyone should learn an additional programming language on top of memorising every standard Unix/GNU/BSD command and their flags.
- LukeShu 13y agoWhat about this? <$file grep "patter" It keeps the flow of information left-to-right, but also avoids UUOC.
- mzs 13y agoAre you sure about that, what if file contains the equal sign? $ touch foo=bar $ file foo=bar foo=bar: empty $ awk '/bar/ { print $1 | "sort -u"}' foo=bar bar screw this! ^D bar $ awk '/bar/ { print $1 | "sort -u"}' <foo=bar $
- kylec 13y agoMentally, I think of most Unix utilities as operating on STDIN and producing STDOUT. Therefore, to get the lines from a file into STDIN, I usually cat it. It's pretty common for me to write something like: cat file | grep xyz Of course, this is completely unnecessary and a "waste" of using cat, as grep accepts as input a file or list of files to operate on, as do many other utilities. However, I don't care. It's additional mental clutter to have to remember the way to specify input from a file in addition to stdin when I know I can always cat the file and pipe it to whatever command I want.
- Amadou 13y agoI'm going to agree with you here. 99% of the time this UUOC stuff is just premature optimization. If it gets the job done it does not matter if it wastes a few milliseconds.
- joshbaptiste 13y agoThe problem is when I see UUOC use in fairly large scripts. I've seen load averages shoot through the roof due to numerous separate commands that are being run causing needless context switching as each set of commands is being run per line of a large datafile.
- dpcx 13y agoFixing it at that point is no longer a "premature optimization"...
- e12e 13y agoWait, what? You mean like this?: cat somefile |while read line do echo $line|cat -|pipe... done Because otherwise I can't see how a pipeline would be started for each line of a datafile ?
- adestefan 13y agoIt's more common to see this issues with something like this: for f in *; do cat $f | ... done Now make that a directory with thousands of files or a script digs down through through the tree.
- emhart 13y agoI felt painfully ignorant when I realized this was nothing to do with felines.
- 16s 13y agocat file | grep string It's a gratuitous use of cat, it's not useless. gratuitous - uncalled for; lacking good reason; unwarranted grep string file Is all you need.
- deleted 13y ago[deleted]
- salmonellaeater 13y agoAnother handy bit of syntax: $ command-name 2>&1 | less This combines standard out and standard error into one stream so you can read all the output in less [1]. Piping just standard out leads to stray error lines popping up on the screen while less is running, and they aren't kept in less's buffer so you lose track of them. [1] http://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/ http://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
- viraptor 13y agoShorter version in bash: |& less It redirects both outputs at once.
- kbenson 13y agoBe careful when redirecting STDERR to STDOUT if yo uare also planning to redirect STDOUT to a file. The order in which things are done affects the outcome. These are not the same: # Redirect STDOUT to file, and STDERR to file command >/tmp/command.out 2>&1 # Redirect STDOUT to file, and STDERR to screen (as STDOUT) command 2>&1 >/tmp/command.out The important thing to remember is that 2>&1 is redirecting to what STDOUT is currently pointing to, not redirecting to STDOUT itself.
- scbrg 13y agoThank you for posting that. I always knew the two behaved differently, and how - but I never really understood why. Your final line cleared that up.
- TorKlingberg 13y agoRecent version of bash support the (much nicer) csh syntax: command >& /tmp/command.out
- kbenson 13y agoI wanted to mention that in the original post, but I can never remember the syntax and was rushed. :(
- alexkus 13y agoMy favourite seemingly-useless-but-actually-not-useless use of cat:- ls | cat
- sophacles 13y agoWhy is it not actually useless?
- ben0x539 13y agoIt makes the output for ls not a terminal, so ls won't try to present the output in a nice human-readable format, with columns and colors etc. Of course there's switches for all that in ls somewhere, but still...
- pdw 13y agoOr just ls -1 (that's a digit one)
- alexkus 13y agoAh, but with the --color=auto option on ls (which I have set in my aliases) there's a difference between "ls -1" and "ls | cat" Indeed there's even a difference between "ls -1" and "ls -1 | cat" if "--color=auto" is specified.
- kpreid 13y agoI use this for programs which otherwise output to pagers, when I'd rather have the output just in terminal scrollback. (And unlike the -1 option it's generic to all programs with terminal behaviors.)
- brokenparser 13y agoFuck Google for the link rot, everything that points to Deja News is now useless. When will they start remembering their own corporate motto?
- skylan_q 13y agoApparently, everything you do in bash is useless unless you're playing bash golf. Nevermind the fact that the commands still output what you want.
- lowmagnet 13y agoThough I've read this before, I've never picked up on the fact that you can do: > <file command It makes sense because the first statement is really saying "set up an input pipe" and not "This points at the command to my left."
- otikik 13y agoUseless Use of Shell Award When there are perfectly valid directives in your language, don't just shell out. We ruby people do this too often.
- bcoates 13y agoI will accept UUOC awards only from people who never use rsync when tar or cp will do.
- sleepydog 13y agoThe "for f in `ls * `; do" example is actually not useless. It has the quality that the loop will not run if there are no files, because ls produces no output. Compare: $ for i in `ls * 2>/dev/null` do cat $i done $ $ for i in * do cat $i done cat: *: No such file or directory Imagine that instead of cat we have a large, complex, expensive loop and don't want to pass a useless '*' to it if we can avoid it.
- dimitar 13y agoThe fact that the same thread ("but but but, I think it's cleaner / nicer / not that much of a waste / my privelege to waste processes!") springs up virtually every time the Award is posted is also Ancient Usenet Tradition.