5 ms·
Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT. Therefore, to get the lines from a file into STDIN, I usually cat it. It's
by kylec 13y ago
Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT. Therefore, to get the lines from a file into STDIN, I usually cat it. It's pretty common for me to write something like:
cat file | grep xyz
Of course, this is completely unnecessary and a "waste" of using cat, as grep accepts as input a file or list of files to operate on, as do many other utilities. However, I don't care. It's additional mental clutter to have to remember the way to specify input from a file in addition to stdin when I know I can always cat the file and pipe it to whatever command I want.
- Amadou 13y agoI'm going to agree with you here. 99% of the time this UUOC stuff is just premature optimization. If it gets the job done it does not matter if it wastes a few milliseconds.
- joshbaptiste 13y agoThe problem is when I see UUOC use in fairly large scripts. I've seen load averages shoot through the roof due to numerous separate commands that are being run causing needless context switching as each set of commands is being run per line of a large datafile.
- dpcx 13y agoFixing it at that point is no longer a "premature optimization"...
- e12e 13y agoWait, what? You mean like this?: cat somefile |while read line do echo $line|cat -|pipe... done Because otherwise I can't see how a pipeline would be started for each line of a datafile ?
- adestefan 13y agoIt's more common to see this issues with something like this: for f in *; do cat $f | ... done Now make that a directory with thousands of files or a script digs down through through the tree.
- e12e 13y agoBut it's not really the "cat" that's the problem here, is it?
- cnvogel 13y agoSure it's possible without running a subprocess, just open the file on some other filedescriptor. exec 5<words.txt while read line <&5 ; do echo "line read: $line" done exec <&5-
- Amadou 13y ago(1) Like dpcx said, dealing with that isn't premature optimization. (2) Be careful about load averages, lots of people don't understand them. It just means the number of processes ready to run over a period of time. It is entirely possible to have a very high load average with a very low cpu utilization. A bunch of little processes that hop on and off the run queue really quickly because they only have a little bit of work to do at a time can kick the load average up way higher than a single self-contained process doing the same amount of work with the same amount of total cpu utilization. The single process wins on context switches, but modern cpus cache those in hardware so its barely a win any more.
- deleted 13y ago[deleted]
- mjb 13y agoExactly. Things like the "useless use of wc -l" fall into a similar category for me. When I want to count the lines of stdout, I use "wc -l". It may not be the simplest, most efficient or fewest keystrokes, but it is extremely easy to remember and fits well into the model of a pipeline of simple single-purpose tools. I tend to use cat at the beginning of pipelines just because it keeps the flow left-to-right, and makes it much clearer what the source of the data is. There's value in knowing some of the advanced things that the textutils and bash can do together, and sometimes there's value in using those things. I don't dispute that, but I do dispute the need to shame people who use the tools in ways that you personally don't like. Shame or not, I'm going to keep using cat at the beginning of my pipelines unless I have a really good reason not to. If that makes some people think I'm not in the Unix wizard in-group, I don't really care.
- sliverstorm 13y agoPipes also match my thought flow. "wc -l" is a good example. "Ok, I want to read this file..." cat file "Then I want to find foo" cat file | grep foo "Oh, and I want to know how many lines of foo there are" cat file | grep foo | wc -l In this trivial example, I could have inserted "-c" with only a few keystrokes, but on a 200 character string of commands it's faster and easier to tack "wc -l" on at the end. Plus I'm not strong enough with grep to do every manipulation I might want, so sometimes that grep will be followed by some perl, or a sort, etc.
- mindcrime 13y agoFWIW, that's the exact same way I approach that sort of manipulation. Start with the most basic operation, then construct a pipeline going left to right, adding filters / operations as needed to get to the result.
- scjody 13y agoIf you're doing a one-off pipeline on the command line, you can even start with less! "I want to look through this file..." less file "Oh, that's long, I want to look at all the foos..." less file | grep foo "How many foos are there?" less file | grep foo | wc -l Try it, it works!
- gav 13y agoAs an alternative, you can always rewrite: cat file | grep xyz To: <file grep xyz
- e12e 13y agoNo? Maybe you meant "grep xyz <file" ? In what shell does your line accomplish the same thing? [edit: Ah, I had an extra pipe in there. "<file command args" does indeed work --- "<file | command args" does not]
- abecedarius 13y agoJust a cute tangent: I once wrote a shell with no built-in redirection. You executed a program named "<" for that, like your second example but with a space: < file grep xyz (The trouble with this design: it doesn't generalize well to pipes.)
- _delirium 13y agoThis form is also nicely modular, setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source: cat file | grep xyz gzcat file | grep xyz xzcat file | grep xyz pv file | grep xyz curl url | grep xyz There are many places I get data, and it sometimes changes (e.g. I decide to compress a file and read it back with xzcat instead of cat, leaving the rest of the script unchanged). Unless you give me an alternate syntax that can handle all of them, I'm not interested in the fact that a particular utility has chosen to provide its own idiosyncratic syntax to special-case only the 'cat' data source, with an -f parameter or whatever.
- LukeShu 13y ago> setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source: Just replace cat file | with <file It still keeps the data source on the left, and easily swapped out: <file grep xyz
- maffydub 13y agoI agree, and like the fact that you can then read your commandline as a series of steps, going from input to output. However, in bash (and other shells?) you can write "<file grep xyz" rather than "cat file | grep xyz".
- deleted 13y ago[deleted]
- pjungwir 13y agoYou could argue that people who complain about wasting a cat are ignoring the Unix philosophy of "do one thing and do it well." Moreso with wc -l vs grep -c. If I know that wc -l is how to count lines, I don't want to remember the options for counting lines built into every other command.
- asdasf 13y ago>Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT Yeah, that is actually very much the point. You are passing the contents of the file in to grep's STDIN. So why not just do exactly that rather than cat it first for no reason? You don't need to remember how to tell commands to read from a file, you just need to learn the basics of how your shell works. <file grep xyz