4 ms·

Somewhat tangential, but the author calls out using `grep` in conjunction with `awk`. Anytime I find myself doing this, it usually turns out that I can just thr

Somewhat tangential, but the author calls out using `grep` in conjunction with `awk`. Anytime I find myself doing this, it usually turns out that I can just throw the pattern into AWK itself. grep 'foo' file.txt | awk '{ print $1 }' Becomes: awk '/foo/ { print $1 }' file.txt There may be times when `grep` is preferable, but this is ubiquitous enough that it's mentioned in the "Useless Use of Cat" [1] awards. [1] https://porkmail.org/era/unix/award https://porkmail.org/era/unix/award

aasasd 5y ago
Both the ‘useless use of cat’ and this nitpick miss that a unix-style command line is very much like a bunch of functions feeding results into each other, with each doing its own thing. This allows one to chain different functions in the same general way: namely, if I want to limit the input at first for debugging, I can do `cat stuff | head -10 | rest-of-commands`, or `grep stuff | head -10 | awk`—and I won't need to keep dragging the `<stuff` input around between commands, or to move the regex from awk to grep and back again or put the limit into the neatly written awk command.
- xyzwave 5y ago
  Agreed, this is one the primary principles of the Unix Philosophy [1]. I was curious how performance would be impacted, and for a large file `grep` may not be so useless. $ du -h /tmp/file.json 161M /tmp/file.json $ time awk -F: '/"id"/ { print $1 }' /tmp/file.json >/dev/null real 0m17.810s user 0m17.690s sys 0m0.083s $ time (grep '"id"' /tmp/file.json | awk -F: '{ print $1 }' >/dev/null) real 0m3.617s user 0m3.641s sys 0m0.037s While pushing the filter into AWK is a bit easier on the eyes, it appears there is an incurred performance cost. [1] http://www.catb.org/~esr/writings/taoup/html/ch01s06.html http://www.catb.org/~esr/writings/taoup/html/ch01s06.html
  - dima55 5y ago
    Can you share the file.json? I'd like to understand this difference better.
    - xyzwave 5y ago
      Unfortunately I picked something from my day job with a bunch of proprietary data. It was essentially a large list of pretty-printed objects with about 20 attributes each. A few of which had multi-paragraph string values.
  - aasasd 5y ago
    You might get an even better speedup by using `grep -F` or ripgrep.
  - cle 5y ago
    Yeah grep will generally be faster than awk for finding patterns in files, since that's the only thing it does, and it can (and does) take advantage of those constraints to apply performance optimizations. Personally I don't think either approach is "better". Sometimes it's easier to deal with some complex logic if it's all in a single awk program, rather than smeared across combinations of grep, awk, and other things. If that outweighs the perf drop in some specific situation, then I'll do it.
- czx4f4bd 5y ago
  This is one of those things that really irks me about the Unix shell. You're totally right that commands like `cat *.txt | grep foo` look right and feel intuitive, but the problem is that it's not actually equivalent to `grep foo *.txt`. With a single input file, it's probably not an issue, but with multiple files the downstream command can't actually see what file it's reading from, so in this example grep can't report filenames with matches and you can't use flags like --include/--exclude based on filenames. I keep wishing for a Unix-like shell and utilities where `cat *.txt | grep foo` actually works the same as `grep foo *.txt`. PowerShell has pulled this off to some extent, which is cool, but it doesn't quite feel right to me compared to Bash.

aasasd 5y ago

Both the ‘useless use of cat’ and this nitpick miss that a unix-style command line is very much like a bunch of functions feeding results into each other, with each doing its own thing. This allows one to chain different functions in the same general way: namely, if I want to limit the input at first for debugging, I can do `cat stuff | head -10 | rest-of-commands`, or `grep stuff | head -10 | awk`—and I won't need to keep dragging the `<stuff` input around between commands, or to move the regex from awk to grep and back again or put the limit into the neatly written awk command.

xyzwave 5y ago
Agreed, this is one the primary principles of the Unix Philosophy [1]. I was curious how performance would be impacted, and for a large file `grep` may not be so useless. $ du -h /tmp/file.json 161M /tmp/file.json $ time awk -F: '/"id"/ { print $1 }' /tmp/file.json >/dev/null real 0m17.810s user 0m17.690s sys 0m0.083s $ time (grep '"id"' /tmp/file.json | awk -F: '{ print $1 }' >/dev/null) real 0m3.617s user 0m3.641s sys 0m0.037s While pushing the filter into AWK is a bit easier on the eyes, it appears there is an incurred performance cost. [1] http://www.catb.org/~esr/writings/taoup/html/ch01s06.html http://www.catb.org/~esr/writings/taoup/html/ch01s06.html
- dima55 5y ago
  Can you share the file.json? I'd like to understand this difference better.
  - xyzwave 5y ago
    Unfortunately I picked something from my day job with a bunch of proprietary data. It was essentially a large list of pretty-printed objects with about 20 attributes each. A few of which had multi-paragraph string values.
- aasasd 5y ago
  You might get an even better speedup by using `grep -F` or ripgrep.
- cle 5y ago
  Yeah grep will generally be faster than awk for finding patterns in files, since that's the only thing it does, and it can (and does) take advantage of those constraints to apply performance optimizations. Personally I don't think either approach is "better". Sometimes it's easier to deal with some complex logic if it's all in a single awk program, rather than smeared across combinations of grep, awk, and other things. If that outweighs the perf drop in some specific situation, then I'll do it.
czx4f4bd 5y ago
This is one of those things that really irks me about the Unix shell. You're totally right that commands like `cat *.txt | grep foo` look right and feel intuitive, but the problem is that it's not actually equivalent to `grep foo *.txt`. With a single input file, it's probably not an issue, but with multiple files the downstream command can't actually see what file it's reading from, so in this example grep can't report filenames with matches and you can't use flags like --include/--exclude based on filenames. I keep wishing for a Unix-like shell and utilities where `cat *.txt | grep foo` actually works the same as `grep foo *.txt`. PowerShell has pulled this off to some extent, which is cool, but it doesn't quite feel right to me compared to Bash.