4 ms·
Agreed, this is one the primary principles of the Unix Philosophy [1]. I was curious how performance would be impacted, and for a large file `grep` may not be
by xyzwave 5y ago
Agreed, this is one the primary principles of the Unix Philosophy [1].
I was curious how performance would be impacted, and for a large file `grep` may not be so useless.
$ du -h /tmp/file.json
161M /tmp/file.json
$ time awk -F: '/"id"/ { print $1 }' /tmp/file.json >/dev/null
real 0m17.810s
user 0m17.690s
sys 0m0.083s
$ time (grep '"id"' /tmp/file.json | awk -F: '{ print $1 }' >/dev/null)
real 0m3.617s
user 0m3.641s
sys 0m0.037s
While pushing the filter into AWK is a bit easier on the eyes, it appears there is an incurred performance cost.
[1] http://www.catb.org/~esr/writings/taoup/html/ch01s06.html http://www.catb.org/~esr/writings/taoup/html/ch01s06.html
- dima55 5y agoCan you share the file.json? I'd like to understand this difference better.
- xyzwave 5y agoUnfortunately I picked something from my day job with a bunch of proprietary data. It was essentially a large list of pretty-printed objects with about 20 attributes each. A few of which had multi-paragraph string values.
- aasasd 5y agoYou might get an even better speedup by using `grep -F` or ripgrep.
- cle 5y agoYeah grep will generally be faster than awk for finding patterns in files, since that's the only thing it does, and it can (and does) take advantage of those constraints to apply performance optimizations. Personally I don't think either approach is "better". Sometimes it's easier to deal with some complex logic if it's all in a single awk program, rather than smeared across combinations of grep, awk, and other things. If that outweighs the perf drop in some specific situation, then I'll do it.