4 ms·

Blog post begins with cat abuse. Fyi, grep [ex] [file] is equivalent to cat [file] | grep [ex]. I get that they needed a simple pipe example

asicsp 4y ago
cat /usr/share/dict/words | grep -e '^a..$' | wc -l can be shortened to grep -cx 'a..' /usr/share/dict/words but yeah, author probably just wanted to show pipe example. I usually show `du -sh -- * | sort -h` for such cases.
- googlryas 4y ago
  Performing 100 of the cat versions on M1 MBP with the default zsh consistently takes between 6.1s and 6.2s, whereas 100 of the grep -cx version consistently takes between 9.1s and 9.3s grep -cx version: for i in {1..100} do grep -cx 'a..' /usr/share/dict/words done cat version: for i in {1..100} do cat /usr/share/dict/words | grep -e "^a..$" | wc -l done (adding a | tr -d " " to the cat version to make the output equivalent only adds about .1s to the total run).
  - asicsp 4y ago
    6 and 9 seconds!! Something seems off. Here's what I get (Linux, Intel Gold G6400 CPU @ 4.00GHz): $ time for i in {1..100}; do grep -cx 'a..' /usr/share/dict/words; done > f1 real 0m0.249s user 0m0.207s sys 0m0.046s $ time for i in {1..100}; do cat /usr/share/dict/words | grep -e '^a..$' | wc -l; done > f2 real 0m0.306s user 0m0.386s sys 0m0.141s Edit: probably because your word file is similar to the one in the post? I've got only 39 matching lines from a 102401 line word file.
    - googlryas 4y ago
      235886 lines with 90 matches on whatever is shipped with macos - shasum /usr/share/dict/words = a62edf8685920f7d5a95113020631cdebd18a185 I also put each version in a separate file, so my time commands were time ~/a and time ~/b, and I let the output go to stdout Also, weirdly, the commands you provided don't print the results from the time command - I'm not sure why (I never use zsh, it's just on the computer I'm using at the moment).
      - asicsp 4y ago
        I just tried with 662349 line file having 177 matches ( got the word file from http://app.aspell.net/create http://app.aspell.net/create). I got `0m1.327s` for just grep command and `0m1.412s` for cat+grep+wc version. I use bash, so not sure about zsh. 6+ seconds really seems off.
        planede 4y ago
        AFAIK GNU grep is a different beast than other grep implementations.
  - eddyg 4y ago
    This is an unfair comparison because you aren't using the same `grep` options. What your test shows is that `-cx` is not as efficient. On an x86 MBP: $ time (for i in {1..100}; {grep -cx 'a..' /usr/share/dict/words}) >/dev/null user=9.22s system=0.34s cpu=94% total=10.097 $ time (for i in {1..100}; {grep -e '^a..$' /usr/share/dict/words | wc -l}) >/dev/null user=6.14s system=0.52s cpu=95% total=6.948 $ time (for i in {1..100}; {cat /usr/share/dict/words | grep -e '^a..$' | wc -l}) >/dev/null user=6.29s system=0.80s cpu=100% total=7.024
    - googlryas 4y ago
      I just used the grep command that OP mentioned in their cat abuse post.
      - taskforcegemini 4y ago
        seems that OPs suggestion was only meant to make the command more concise
kqr 4y ago
Not equivalent. If you want to insert an intermediary processing stage between the input file and the grep, the initial cat makes that a mechanical insertion, mirroring the intention in syntax. Without the cat, it's a more complex transformation.
- loa_in_ 4y ago
  Additionally it costs almost nothing to run a cat, but some mental work to remember which programs are built to prevent it (accept filename argument). If they are one can pass /proc/self/fd/0 as filename... That's some mindless tinkering that works so universally that it's just practical
  - ithkuil 4y ago
    You can reduce the cognitive overhead by using this pattern: Before: <input.txt sed ... | awk ... After: <input.txt grep bar | sed ... | awk .. Every new pipeline stage is just "command args |" and you insert it after the end of a previous command, treating the initial input redirect as the end of zeroth command
    - Izkata 4y ago
      ...that increases the mental overhead because now there's no delimiter between the first and second part of the pipeline.
      - ithkuil 4y ago
        in zsh you can do this: <input.txt | grep bar | sed ... | awk .. but not in bash
  - deleted 4y ago
    [deleted]
- samuell 4y ago
  Exactly this. I.e. I often back and forth comment out parts of a pipeline, when debugging. Thus, with `cat [file] | grep [ex]` I could do: cat [file] | less # | grep [ex] ... to quickly check the output of the unprocessed data, before uncommenting again. This is not possible with `grep [ex] [file]`. The power of this of course shines the most when having really long pipes, of up to or over 10 parts (which I've had lots of in my career). (Addendum: Well, I of course actually have an alias s='less -S' to make this even more fluent, so I do: cat [file] | s # | grep [ex] ... to peek into various parts of a pipeline. -S is for not folding long lines.)
elchief 4y ago
Ya,ya. Cat abuse is often mistaken for "I'm using cat as a placeholder for some other more complicated process". Get over it
- topspin 4y ago
  There have been times recently when I've used cat in places where an alternative exists just in the hope that it annoys someone indulging this hang-up.
  - bravetraveler 4y ago
    pv may fit the bill, I use it to get transfer rates for disk images I'm writing ref: http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml
- kjs3 4y ago
  Indeed. I'm genuinely baffled that anyone would care enough to mention it given how common an idiom it is. When did this so-called 'cat abuse' rise to the level of 'I must chastise this person for their sin'?
  - fragmede 4y ago
    it's a micro optimization and some people see all of life as an optimization problem.
    - kjs3 4y ago
      Insightful. And rather sad. I suspect there's more than a slice of "old == bad" mixed up in this attitude as well. P.S. - do not search for 'cat abuse' to understand this mentality. Rookie error.
      - fragmede 4y ago
        https://porkmail.org/era/unix/award https://porkmail.org/era/unix/award is a good reference if you want to read what the heretics believe about cat and kill and wc -l.
dxuh 4y ago
I think the root cause of cat abuse is that grep just takes the parameters the wrong way around. It happens much more often that I want to tweak the pattern and I know which file(s) to grep than the other way around.
- mcbits 4y ago
  I think it's less that the parameters are the wrong way around and more that the file-reading feature in grep is redundant and could be deleted.
  - planede 4y ago
    Except when you pass `-r` and you pass a directory in place of the filename. But maybe it should have been `-r <directory>` and still no positional file parameter. The ship has been sailed a long ago though.
    - mprovost 4y ago
      The newer generation of grep-like utilities (ack, ag, rg) recurse through a directory parameter by default. Of course you could always do this with find.
mejutoco 4y ago
Anybody knows if some shells apply these easy optimizations? Could a shell detect cat abuse and replace it with the right grep incantation?
- thewakalix 4y ago
  cat and grep generally aren't builtins, so shells can't rely on them to have any particular behavior.
- gpderetta 4y ago
  Cat could use splice and basically reduce the overhead to nothing. It possibly already does.
- MisterTea 4y ago
  > Could a shell detect cat abuse and replace it with the right grep incantation? That sounds like shell abuse via bloat.
  - mejutoco 4y ago
    Not saying it is necessarily a good idea, just curious if any such optimization is possible. It seems (from thewakalix comment above) not being a built in it would not be possible. Now we only need to remove the other 100% of the bloat in the shell ;)

asicsp 4y ago

cat /usr/share/dict/words | grep -e '^a..$' | wc -l can be shortened to grep -cx 'a..' /usr/share/dict/words but yeah, author probably just wanted to show pipe example. I usually show `du -sh -- * | sort -h` for such cases.

googlryas 4y ago
Performing 100 of the cat versions on M1 MBP with the default zsh consistently takes between 6.1s and 6.2s, whereas 100 of the grep -cx version consistently takes between 9.1s and 9.3s grep -cx version: for i in {1..100} do grep -cx 'a..' /usr/share/dict/words done cat version: for i in {1..100} do cat /usr/share/dict/words | grep -e "^a..$" | wc -l done (adding a | tr -d " " to the cat version to make the output equivalent only adds about .1s to the total run).
- asicsp 4y ago
  6 and 9 seconds!! Something seems off. Here's what I get (Linux, Intel Gold G6400 CPU @ 4.00GHz): $ time for i in {1..100}; do grep -cx 'a..' /usr/share/dict/words; done > f1 real 0m0.249s user 0m0.207s sys 0m0.046s $ time for i in {1..100}; do cat /usr/share/dict/words | grep -e '^a..$' | wc -l; done > f2 real 0m0.306s user 0m0.386s sys 0m0.141s Edit: probably because your word file is similar to the one in the post? I've got only 39 matching lines from a 102401 line word file.
  - googlryas 4y ago
    235886 lines with 90 matches on whatever is shipped with macos - shasum /usr/share/dict/words = a62edf8685920f7d5a95113020631cdebd18a185 I also put each version in a separate file, so my time commands were time ~/a and time ~/b, and I let the output go to stdout Also, weirdly, the commands you provided don't print the results from the time command - I'm not sure why (I never use zsh, it's just on the computer I'm using at the moment).
    - asicsp 4y ago
      I just tried with 662349 line file having 177 matches ( got the word file from http://app.aspell.net/create http://app.aspell.net/create). I got `0m1.327s` for just grep command and `0m1.412s` for cat+grep+wc version. I use bash, so not sure about zsh. 6+ seconds really seems off.
      - planede 4y ago
        AFAIK GNU grep is a different beast than other grep implementations.
- eddyg 4y ago
  This is an unfair comparison because you aren't using the same `grep` options. What your test shows is that `-cx` is not as efficient. On an x86 MBP: $ time (for i in {1..100}; {grep -cx 'a..' /usr/share/dict/words}) >/dev/null user=9.22s system=0.34s cpu=94% total=10.097 $ time (for i in {1..100}; {grep -e '^a..$' /usr/share/dict/words | wc -l}) >/dev/null user=6.14s system=0.52s cpu=95% total=6.948 $ time (for i in {1..100}; {cat /usr/share/dict/words | grep -e '^a..$' | wc -l}) >/dev/null user=6.29s system=0.80s cpu=100% total=7.024
  - googlryas 4y ago
    I just used the grep command that OP mentioned in their cat abuse post.
    - taskforcegemini 4y ago
      seems that OPs suggestion was only meant to make the command more concise

kqr 4y ago

Not equivalent. If you want to insert an intermediary processing stage between the input file and the grep, the initial cat makes that a mechanical insertion, mirroring the intention in syntax. Without the cat, it's a more complex transformation.

loa_in_ 4y ago
Additionally it costs almost nothing to run a cat, but some mental work to remember which programs are built to prevent it (accept filename argument). If they are one can pass /proc/self/fd/0 as filename... That's some mindless tinkering that works so universally that it's just practical
- ithkuil 4y ago
  You can reduce the cognitive overhead by using this pattern: Before: <input.txt sed ... | awk ... After: <input.txt grep bar | sed ... | awk .. Every new pipeline stage is just "command args |" and you insert it after the end of a previous command, treating the initial input redirect as the end of zeroth command
  - Izkata 4y ago
    ...that increases the mental overhead because now there's no delimiter between the first and second part of the pipeline.
    - ithkuil 4y ago
      in zsh you can do this: <input.txt | grep bar | sed ... | awk .. but not in bash
- deleted 4y ago
  [deleted]
samuell 4y ago
Exactly this. I.e. I often back and forth comment out parts of a pipeline, when debugging. Thus, with `cat [file] | grep [ex]` I could do: cat [file] | less # | grep [ex] ... to quickly check the output of the unprocessed data, before uncommenting again. This is not possible with `grep [ex] [file]`. The power of this of course shines the most when having really long pipes, of up to or over 10 parts (which I've had lots of in my career). (Addendum: Well, I of course actually have an alias s='less -S' to make this even more fluent, so I do: cat [file] | s # | grep [ex] ... to peek into various parts of a pipeline. -S is for not folding long lines.)

elchief 4y ago

Ya,ya. Cat abuse is often mistaken for "I'm using cat as a placeholder for some other more complicated process". Get over it

topspin 4y ago
There have been times recently when I've used cat in places where an alternative exists just in the hope that it annoys someone indulging this hang-up.
- bravetraveler 4y ago
  pv may fit the bill, I use it to get transfer rates for disk images I'm writing ref: http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml
kjs3 4y ago
Indeed. I'm genuinely baffled that anyone would care enough to mention it given how common an idiom it is. When did this so-called 'cat abuse' rise to the level of 'I must chastise this person for their sin'?
- fragmede 4y ago
  it's a micro optimization and some people see all of life as an optimization problem.
  - kjs3 4y ago
    Insightful. And rather sad. I suspect there's more than a slice of "old == bad" mixed up in this attitude as well. P.S. - do not search for 'cat abuse' to understand this mentality. Rookie error.
    - fragmede 4y ago
      https://porkmail.org/era/unix/award https://porkmail.org/era/unix/award is a good reference if you want to read what the heretics believe about cat and kill and wc -l.

dxuh 4y ago

I think the root cause of cat abuse is that grep just takes the parameters the wrong way around. It happens much more often that I want to tweak the pattern and I know which file(s) to grep than the other way around.

mcbits 4y ago
I think it's less that the parameters are the wrong way around and more that the file-reading feature in grep is redundant and could be deleted.
- planede 4y ago
  Except when you pass `-r` and you pass a directory in place of the filename. But maybe it should have been `-r <directory>` and still no positional file parameter. The ship has been sailed a long ago though.
  - mprovost 4y ago
    The newer generation of grep-like utilities (ack, ag, rg) recurse through a directory parameter by default. Of course you could always do this with find.

mejutoco 4y ago

Anybody knows if some shells apply these easy optimizations? Could a shell detect cat abuse and replace it with the right grep incantation?

thewakalix 4y ago
cat and grep generally aren't builtins, so shells can't rely on them to have any particular behavior.
gpderetta 4y ago
Cat could use splice and basically reduce the overhead to nothing. It possibly already does.
MisterTea 4y ago
> Could a shell detect cat abuse and replace it with the right grep incantation? That sounds like shell abuse via bloat.
- mejutoco 4y ago
  Not saying it is necessarily a good idea, just curious if any such optimization is possible. It seems (from thewakalix comment above) not being a built in it would not be possible. Now we only need to remove the other 100% of the bloat in the shell ;)