15 ms·

Skip grep, use awk

• netheril96 9y ago
  Isn't rg the hot one for grep replacement these days?
  • hayd 9y ago
    can't go back, ripgrep is so fast.
  • clishem 9y ago
    Took me a while to understand the name.
  • clumsysmurf 9y ago
    rg + fzf = <3
  • Sean1708 9y ago
    It doesn't replace Awk though. I think the point of this post is that you shouldn't do `<search tool> <pattern> | awk ...`, you should just do `awk /<pattern>/ ...`.
• kiwidrew 9y ago
  While awk is indeed under-appreciated, there are many instances where using grep to pre-filter your input is helpful because the grep family (GNU grep, agrep, etc.) can match strings much much faster than awk's regex engine. For example: GNU grep uses a heavily optimized implementation of the Boyer-Moore string search algorithm [1] which (it is claimed) requires just 3 (x86) cycles per byte of input. Boyer-Moore only searches for literal strings, but grep will extract the longest literal from the search regex (e.g. the string "foo" from the regex /foo(bar|baz)+/) and use Boyer-Moore to find candidate matches that are then verified using the regex engine. So if you have a large corpus of text to search, grep is your friend. It can easily saturate your I/O bandwidth and perform an order of magnitude faster than typical "big data" platforms. [2] [1] https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... [2] https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
  • voltagex_ 9y ago
    I have found ack and ag to be much faster than grep (I should actually time this).
    • thesmallestcat 9y ago
      They are perceived as faster because they automatically skip binary and VCS-ignored files. grep is faster.
      • lathiat 9y ago
        They are faster for practical usage, though I'm fairly sure ag also executes the search in parallel no? I also seem to recall the ripgrep author recently trying to optimise towards grep in another HN post. Love ag ever since I discovered it, though you have to be careful every so often it doesn't look in a filetype that mattered.
        • deleted 9y ago
          [deleted]
      • petdance 9y ago
        > they automatically skip binary and VCS-ignored files If it does less work, and takes less time, then isn't that faster?
    • TheDong 9y ago
      See http://blog.burntsushi.net/ripgrep/ http://blog.burntsushi.net/ripgrep/ for a quite nice comparison which is counter to your experience; ack/ag/pt are all slower than either grep or ripgrep.
      • burntsushi 9y ago
        "ack/ag/pt are all slower than either grep or ripgrep" You need to be a touch careful with blanket statements, as this conclusion isn't quite what the data in my blog says. What it says is that ag is generally slower than GNU grep on large files, primarily because GNU grep's core search code is quite a bit more optimized than ag's. However, ag can outclass GNU grep when searching across entire directories primarily by culling the set of files that is searched, and also by parallelizing search if you want to exclude running GNU grep in a simple `find ... | xargs -P`-like command. (This is thesmallestcat's point.) This is why the first set of benchmarks in my blog don't actually benchmark GNU grep, because there is an impedance mismatch. Instead, ag/pt are benchmarked against `git grep`, where the comparison is quite a bit closer. (You'll want to check out the benchmarks on my local machine, which avoid penalizing ag for its use of memory maps.[1]) The second set of benchmarks basically swap out `git grep` for GNU grep and compare the tools on very large files. ack isn't actually included in the benchmarks because it was incredibly slow when I tried it, although there may be something funny going on.[2] To be honest, it isn't terribly surprising to me, since every other tool is compiled down to native code. The pt/sift story is interesting, and my current hypothesis is that the tools have received a misleading reputation for being fast. In particular, when using the tools with simple literals, it's likely that search will benefit from an AVX2 optimization when searching for the first byte. As soon as you start feeding more complex patterns (including case insensitivity), these tools slow down quite a bit because 1) the underlying regex engine needs more optimization work and 2) the literal analysis is lacking, which means the tools rely even more on the regex engine than other tools do. The short summary of ripgrep is that it should outclass all the tools on all the benchmarks in my blog. I should update them at some point as ripgrep has gotten a bit faster since that blog post. Primarily, a parallel recursive directory iterator, which is present in sift, pt and ucg as well. Secondarily, its line counting has been sped up with more specialized SIMD routines. (I should hedge a bit here. It is possible to build patterns that a FSM-like engine will do very poorly on, but where a backtracking engine may not degrade as much. The prototypical example is large repetitions, e.g., `(foo){100}`. Of course, the reverse is true as well, since FSMs don't have exponential worst case time. Also, my benchmarks aren't quite exhaustive. For example, they don't benchmark GNU grep/ripgrep's `-f` flag for searching many regexes.) [1] - https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2016-09-22-archlinux-cheetah/summary https://github.com/BurntSushi/ripgrep/blob/master/benchsuite... [2] - https://github.com/petdance/ack3/issues/42 https://github.com/petdance/ack3/issues/42
• rhizome 9y ago
  To address one of the benefits, `egrep ^[^#]` is shorter than `awk '/^[^#]/'`.
  • thesmallestcat 9y ago
    That pattern doesn't need extended regular expressions (use grep -E instead of egrep though). Also `grep -v ^#` does the same thing. `sed /^#/d` is one char shorter.
    • vacri 9y ago
      > Also `grep -v ^#` does the same thing. Depends on what you're after. 'grep ^[^#]' also gets rid of empty lines, as they don't have a first character to match.
• emmelaich 9y ago
  If you don't mind a bit of Perl, then Perl can be used for grep, awk and sed. And find. Even comes with little conversion utilities to do it (mostly) for you: a2p, s2p, find2perl. I used Python most of the time but still use Perl where it is appropriate.
  • jedimastert 9y ago
    perl is still one of my favorite programming languages to use for straight text-manipulation
  • jwilk 9y ago
    These converters were actually removed from Perl in v5.21.1. They are available as a separate distributions on CPAN: https://metacpan.org/pod/App::a2p https://metacpan.org/pod/App::a2p https://metacpan.org/pod/App::s2p https://metacpan.org/pod/App::s2p https://metacpan.org/pod/App::find2perl https://metacpan.org/pod/App::find2perl
• emmelaich 9y ago
  My favourite bad example of using grep was from a big enterprise software vendor to kill one of their processes. It looked something like ps -ef| grep SomeDaemon | grep -v grep | grep -v perl | perl -e '<do something with the pid>'
  • lathiat 9y ago
    grep -v grep.. I wish I could admit to using this less often than I do; but it's always manual not automated. At some point I'll figure out a better alternative than this or pgrep which often misses processes.
    • vacri 9y ago
      Just do 'grep [S]omeDaemon', and you won't see your grep line.
      • dozzie 9y ago
        No, please don't do that. Use filtering capabilities of ps, so grep becomes completely unnecessary.
    • majewsky 9y ago
      For manual use, I have a shell function psgrep () { ps aux | sed -n '1p;/\<sed\>/d;/'"$1"'/p' } It's basically the same "grep -v grep" trick, but with pure sed. Also, the initial "1p" ensures that I still get the column headers from ps.
  • thaumaturgy 9y ago
    Serious question: what would be a better way to do this?
    • desdiv 9y ago
      perl -e 'print `pgrep SomeDaemon `'
    • deleted 9y ago
• beefsack 9y ago
  ripgrep[1] is functionally incredibly similar to grep and ag, but is significantly faster[2] and supports a wider range of character encodings. In its short lifetime it has already become the default search tool for VSCode. I've switched to using it as my daily driver for text search and am incredibly happy with it. [1]: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep [2]: http://blog.burntsushi.net/ripgrep/ http://blog.burntsushi.net/ripgrep/ Edit: I confused awk with ag originally leading to this comment. Using ripgrep as a pre-filter to awk is still a ridiculous amount faster, especially on large trees, so while the OP's suggestion is cool I can't see myself reaching for it often.
  • burntsushi 9y ago
    > and supports a wider range of character encodings Since this is an uncommon feature, I'd like to emphasize this. :-) In particular, ripgrep will automatically search UTF-16 encoded files via BOM sniffing. That means you can run ripgrep over a directory on Windows and be confident that it will correctly search both UTF-8 and UTF-16 encoded files automatically without having to think about it. More generally, it supports all encodings found in the Encoding Standard[1]. However, only UTF-16 is automatically detected (where UTF-8 is the presumed default), so you'll need to explicitly specify `-E sjis` (for example) if you want to search Shift_JIS encoded files. Also, I didn't really need to do much of anything to get this working. This is all thanks to @hsivonen's encoding_rs[2] crate, which is now (I think) in Firefox. [1] - https://encoding.spec.whatwg.org/#concept-encoding-get https://encoding.spec.whatwg.org/#concept-encoding-get [2] - https://github.com/hsivonen/encoding_rs https://github.com/hsivonen/encoding_rs
  • cyphar 9y ago
    awk is a full programming language, and most of the times that I'm doing awk scripting I have to use things like associative arrays and arithmetic. Pulling out a field from a line is what most people use awk for, but it's honestly the least interesting part of awk. In fact, if cut supported regular expressions for specifying the field and record separators people wouldn't be using awk for that purpose (because that's all that $n does). I was under the impression that ripgrep is a grep implementation that was incredibly optimised thanks to BurntSushi being a complete madman.
• manasvi_gupta 9y ago
  I have used awk to color application logs (info=green, error=red, warn=yellow) on linux console. Blog - http://manasvigupta.github.io/2015/06/27/color-your-logs-and-stack-traces.html http://manasvigupta.github.io/2015/06/27/color-your-logs-and...
• fiercenoodle 9y ago
  He says that you can't cleanly do grep -v with awk, but awk '!/foo/' seems to work in my copy of awk.
• tonmoy 9y ago
  How am I supposed to do `grep -nrb --include \*.cpp something` ?
  • Xophmeister 9y ago
    This is close, but I don't know how to implement grep's `-b` easily: find . -type f -name "*.cpp" -exec awk '/something/ {print FILENAME, NR, $0}' {} \+ Clearly grep wins this round!
• topspin 9y ago
  But I like my colorized match: grep --color ... Also grep -n is so much nicer than something like awk '{print NR "," $0}' Finally, I think grep must be faster if only because grep doesn't line buffer by default, although you may --line-buffered.
• bluetomcat 9y ago
  AWK is the general-purpose programmatic filter and reporting tool in the Unix pipeline. Sed, grep and cut are specializations for specific use cases whose implementations might have better performance. Perl and Python are probably too general-purpose for writing compact one-liners in a pipeline.
  • rflrob 9y ago
    Even though it's more characters, I usually use perl -pe 's/search/replace/' instead of sed in pipelines because it understands /n (and other escape characters I don't remember). Because all it takes is to get burned a couple times for it to be worth sticking with what you know will work.
    • _delirium 9y ago
      The Perl version is considerably faster on large files as well. Or at least, it is when compared to the version of BSD sed shipped with OSX.
      • kronos29296 9y ago
        This is probably not the case with the sed in Linux.
    • dozzie 9y ago
      "/n"? What's that? My `man perlre' doesn't say anything about this regexp flag.
      • tribby 9y ago
        because OP is referring to escape characters I assume it is a typo for "\n"
      • jwilk 9y ago
        Probably not what rflrob meant, but /n prevents parentheses from capturing. https://perldoc.perl.org/perlre.html#%2an%2a https://perldoc.perl.org/perlre.html#%2an%2a
• kubakuba 9y ago
  Yay, now I've found a replacement for grep under macOS. Thanks for that!
  • _ph_ 9y ago
    Why would you need that, when grep comes with macOS?
    • kubakuba 9y ago
      but this version does not support Perl-like expressions
• dredmorbius 9y ago
  The reason my one-liners often end up with both grep and awk pipes is because that's how they're built: run some command, grep for content of interest, throw awk at it to manipulate output. If this is a throwaway, the exploration flexibility is more important than the command cleanness, and there are some awk idioms (inverted match, case-insensitive, match count, first match only) which are easier to code than their awk equivalents. Sure, if I'm going to commit that to a script or shell function, I'll consider going back to refactor it and clean it up. But for quick-and-dirty exploration and prototyping, a long shell pipe is often the best modular way to compose a tool. Protip: M-X-E will call up your one-liner into an editor session, from which you can save it directly to a permanent file. The number of locally-written tools which have originated in this fashion is ... probably embarassing to admit.
• hiq 9y ago
  The author mentions that using awk instead of grep -v is not a good idea, what about: awk '! /something/' Doesn't it reproduce the same behaviour as the following? grep -v 'something'
  • lkirk 9y ago
    Yes, these are the same
  • gbrown_ 9y ago
    Yup, was going to post this as a comment myself.
• usgroup 9y ago
  Except "grep | awk" is often way quicker because it does not need to split the record before it does anything and I usually want to span more processes because awk will usually eat a whole cpu if it's doing something gnarly.
• torrent-of-ions 9y ago
  This is basic awk. But grep is much faster if that's all you're doing. fgrep even more so if that's what you want.
• thinkMOAR 9y ago
  is there any awk equivalent for grep "matcha\|matchb" ?
  • gbrown_ 9y ago
    awk '/matcha|matchb/'
    • thinkMOAR 9y ago
      excellent :) though slightly disappointed in myself, that i did not try that myself as i did imagine it when i asked...
• assafmo 9y ago
  It's a nice idea, but I think maybe only to replace a simple egrep. grep has tons of great option like -F = fgrep; no regex - way faster -v = as mentioned in the article -o = print only matched input, not the entire line -C = context, print lines before and after the match; can also be used partially with -A (after) and -B (before)
  • tom_mellior 9y ago
    Also, super handy: grep colors the part of the line that matched.
  • burntsushi 9y ago
    Small note: For GNU grep on single regexes at least, the -F flag should not impact performance. It is smart enough to see through a pattern as a literal and avoid the regex engine.
    • assafmo 9y ago
      Thanks, I didn't know this.
      • dullgiulio 9y ago
        Of course you might still want to use fgrep to make sure + and . match with the characters and are not interpreted like regular expression operators.
• jwilk 9y ago
  > awk uses modern (read “Perl”) regular expressions, by default – like grep -E No, Perl regular expression are different than extended regular expressions that awk and "grep -E" use.
• linedash 9y ago
  Use of grep -v equiv is wrong and dirty. Instead should be: $ [data is generated] | awk '!/something/'
• aargh_aargh 9y ago
  I'd like to share a recent experience on a related note, but in the opposite spirit - rather than reduce the number of command invocations on a command line, it may make sense to increase it. I had a loop operating on a text file like this: while read line do echo "$line" | sed -e "s/A/X/" -e "s/B/Y/" -e "s/C/Z/" ... Gradually, as I added more things to replace, I noticed severe slowdown. Things got fast again when I rewrote it as while read line do echo "$line" | sed -e "s/A/X/" | sed -e "s/B/Y/" | sed -e "s/C/Z/" ... Turns out the later (multiple processes in parallel) helped with throughput.
  • klodolph 9y ago
    As a minor note, the first one can be written as "s/A/X;s/B/Y;s/C/Z/"
  • avaika 9y ago
    why do you need `while read` part at all? this is the main slow down here. shell is extremely time consuming compared to sed/awk/whatever tool.
    • aargh_aargh 9y ago
      The script is currently a filter - the input comes from stdin.
      • sebcat 9y ago
        You can redirect stdin to sed directly, e.g., cat << EOF | sed -e 's/A/X/;s/B/Y/;s/C/Z/;' A Brave Cow Was Walking Crows Were Airborne All Was Well EOF or, cat << EOF > myfile A Brave Cow Was Walking Crows Were Airborne All Was Well EOF sed -e 's/A/X/;s/B/Y/;s/C/Z/;' < myfile or (copy-pasted from shell for clarity), $ cat foo.sh #!/bin/sh sed -e 's/A/X/;s/B/Y/;s/C/Z/;' $ cat << EOF | ./foo.sh A Brave Cow EOF X Yrave Zow
        • aargh_aargh 9y ago
• Ultimatt 9y ago
  Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data is generated] | glorp '/Something/ and say $F[2]' We have access to any Perl module by putting -MModule::Name=function after the command, the following will parse a JSON record per line and glorp out what we wanted: $ echo -e '{"hello":"world"}\n{"hello":"cat"}' | glorp 'say decode_json($_)->{hello};' -MJSON=decode_json world cat Maybe you are used to using curl too. There is a nice web framework in Perl called Mojolicious (http://mojolicious.org http://mojolicious.org) that provides a convenience module called 'ojo' for command line use. So grabbing the summary sentence from Wikipedia articles is as straight forward as below. Notice Mojolicious lets us use CSS selectors! $ echo -e 'grep\nawk\nperl' \ | glorp 'say g("wikipedia.org/wiki/$F[0]")->dom->at("#mw-content-text > div > p")->all_text' -Mojo
  • d33 9y ago
    Hey, it would be awesome to have something like this for Python!
    • Ultimatt 9y ago
      Not really possible in Python for anything overly useful/complex because of the significant white space, and regular expressions are not a first class object in the language. ipython is the equivalent solution I suppose. However, Ruby completely inherited this aspect of Perl and is another good candidate. It's just even old systems have a perl installed that will perform the awk like functionality.
      • phaemon 9y ago
        There is "Pyed Piper" aka `pyp` https://code.google.com/archive/p/pyp/ https://code.google.com/archive/p/pyp/ "Pyp is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment."
• skywhopper 9y ago
  awk is certainly a really important tool to know, but in general this is poor advice. awk is significantly slower than grep -E, and typing awk commands is often much slower as well. Not to mention that awk only operates on a single stream of data, and can't do operations with file awareness. Sure, use awk's line filtering when you're already going to need awk for something else, but in general your first instinct should be grep.
• adsche 9y ago
  Just FYI, the code lines do not wrap and are not scrollable for me on mobile (Firefox/Android).
• tannhaeuser 9y ago
  I'm sure the author is aware, but awk has at least three implementations: nawk (the one true), gawk (what most are using), and mawk (performance-oriented, unmaintained). Plus busybox-awk. When benchmarking gawk, I've found using LANG=C and avoiding UTF-8 to make a substantial difference for pattern matching.
  • nisa 9y ago
    > I've found using LANG=C This is also true for grep and tr and sort - the unicode handling does impact speed quite a bit and I still don't understand why this here https://stackoverflow.com/q/20226851/772013 https://stackoverflow.com/q/20226851/772013 is not treated as a bug
  • bsg75 9y ago
    Mawk is currently maintained: http://invisible-island.net/mawk/mawk.html http://invisible-island.net/mawk/mawk.html
• readme 9y ago
  grep -E performs an extended regex. To get a perl compatible one you need to use grep -P.
• racl101 9y ago
  I've always wanted to learn to use awk but I just can never find enough examples that allow me to understand what the hell I'm doing. The learning curve is too high for me.