4 ms·

Really? My completely anecdotal experience is that ag kills grep. Stuff like searching usages of variables in Linux kernel seems to take ages with grep, whereas

by shandor 10y ago

Really? My completely anecdotal experience is that ag kills grep. Stuff like searching usages of variables in Linux kernel seems to take ages with grep, whereas ag search on the whole repository doesn't take annoyingly long. I'll need to test this, just to see how wrong my "feeling" on the matter may be.

ggreer 10y ago
I'm the author of ag. On a single file, ag is slower than GNU grep. The main reason is that ag prints line numbers when showing matches. Printing line numbers means going through the whole file and searching for \n's. For large files, this can be very resource-intensive. Here are the numbers (median of 5 runs, caches hot) for an 8GB file: % time ag hello4294967296 big_file.txt 134217728:hello4294967296 ag hello4294967296 big_file.txt 7.83s user 0.44s system 99% cpu 8.275 total % time grep hello4294967296 big_file.txt hello4294967296 grep hello4294967296 big_file.txt 2.17s user 0.91s system 99% cpu 3.085 total If you enable line numbers with grep -n, grep still wins, probably because ag's printing code is ridiculously inefficient: % time grep -n hello4294967296 big_file.txt 134217728:hello4294967296 grep -n hello4294967296 big_file.txt 3.37s user 0.97s system 99% cpu 4.338 total Unfortunately, I didn't write ag's printing code very sanely. It grew organically from users asking for small features here and there, and now it's a giant hairball. I've added some tests to reduce the chance of breaking changes, but it'll probably be a while before I fix this performance issue. Despite these limitations, ag tends to be much faster than grep -r in real-world use. There are several reasons why: 1. Ag has a multi-threaded architecture that can search multiple files at once. Grep is single-threaded. 2. Ag ignores binary and hidden files by default. Grep doesn't. 3. Ag obeys .gitignore/agignore/hgignore. Grep doesn't. These things really add up. For example, when searching my 20GB ~/code directory: ggreer@lithium:~/code% time ag instantiationname ... ag instantiationname 3.82s user 2.16s system 157% cpu 3.797 total ggreer@lithium:~/code% time grep -n -i -r instantiationname ... grep -n -i -r instantiationname 42.77s user 2.62s system 99% cpu 45.403 total Both find the same matches (some tests in the Node.js source), but ag does it 12x faster.
- burntsushi 10y ago
  > The main reason is that ag prints line numbers when showing matches. Wouldn't it be more accurate to say that ag is slower than grep because it does line-by-line searching where as grep does not? What would happen if ag adopted grep's method of searching where it doesn't do line-by-line? Is that possible with PCRE? I think it would be very interesting to combine the tricks used by grep with the tricks used by ag. I don't think it has been done yet!
  - ggreer 10y ago
    Ag doesn't search line-by-line. It's only if a match is found that it goes back and looks for newlines. This optimizes the typical case (most files have no matches) at the expense of the rare case (file contains a match). Also, ag only uses PCRE if the query looks like a regex. Otherwise, it uses Boyer-Moore strstr.
    - burntsushi 10y ago
      > Ag doesn't search line-by-line. Really? Am I misunderstanding this code that very much appears to search line by line? https://github.com/ggreer/the_silver_searcher/blob/master/src/search.c#L113-L121 https://github.com/ggreer/the_silver_searcher/blob/master/sr... GNU grep never does "throw line into buffer and then search it." GNU grep just searches the entire buffer at once.
      - ggreer 10y ago
        You didn't check to see what value opts.multiline has. It's initialized to TRUE in src/options.c. It's only set to FALSE if a user passes the undocumented --nomultiline option to ag. Default behavior with a regex query is https://github.com/ggreer/the_silver_searcher/blob/master/src/search.c#L92-L111 https://github.com/ggreer/the_silver_searcher/blob/master/sr...
        burntsushi 10y ago
        Ah excellent, thank you for fixing my understanding. :-)
bsg75 10y ago
IIRC, Ag excels at code search, where grep can outpace it when searching things like data files. "The Silver Searcher is a tool for searching code." (http://geoff.greer.fm/ag/ http://geoff.greer.fm/ag/)

by shandor 10y ago

ggreer 10y ago

I'm the author of ag. On a single file, ag is slower than GNU grep. The main reason is that ag prints line numbers when showing matches. Printing line numbers means going through the whole file and searching for \n's. For large files, this can be very resource-intensive. Here are the numbers (median of 5 runs, caches hot) for an 8GB file: % time ag hello4294967296 big_file.txt 134217728:hello4294967296 ag hello4294967296 big_file.txt 7.83s user 0.44s system 99% cpu 8.275 total % time grep hello4294967296 big_file.txt hello4294967296 grep hello4294967296 big_file.txt 2.17s user 0.91s system 99% cpu 3.085 total If you enable line numbers with grep -n, grep still wins, probably because ag's printing code is ridiculously inefficient: % time grep -n hello4294967296 big_file.txt 134217728:hello4294967296 grep -n hello4294967296 big_file.txt 3.37s user 0.97s system 99% cpu 4.338 total Unfortunately, I didn't write ag's printing code very sanely. It grew organically from users asking for small features here and there, and now it's a giant hairball. I've added some tests to reduce the chance of breaking changes, but it'll probably be a while before I fix this performance issue. Despite these limitations, ag tends to be much faster than grep -r in real-world use. There are several reasons why: 1. Ag has a multi-threaded architecture that can search multiple files at once. Grep is single-threaded. 2. Ag ignores binary and hidden files by default. Grep doesn't. 3. Ag obeys .gitignore/agignore/hgignore. Grep doesn't. These things really add up. For example, when searching my 20GB ~/code directory: ggreer@lithium:~/code% time ag instantiationname ... ag instantiationname 3.82s user 2.16s system 157% cpu 3.797 total ggreer@lithium:~/code% time grep -n -i -r instantiationname ... grep -n -i -r instantiationname 42.77s user 2.62s system 99% cpu 45.403 total Both find the same matches (some tests in the Node.js source), but ag does it 12x faster.

burntsushi 10y ago
> The main reason is that ag prints line numbers when showing matches. Wouldn't it be more accurate to say that ag is slower than grep because it does line-by-line searching where as grep does not? What would happen if ag adopted grep's method of searching where it doesn't do line-by-line? Is that possible with PCRE? I think it would be very interesting to combine the tricks used by grep with the tricks used by ag. I don't think it has been done yet!
- ggreer 10y ago
  Ag doesn't search line-by-line. It's only if a match is found that it goes back and looks for newlines. This optimizes the typical case (most files have no matches) at the expense of the rare case (file contains a match). Also, ag only uses PCRE if the query looks like a regex. Otherwise, it uses Boyer-Moore strstr.
  - burntsushi 10y ago
    > Ag doesn't search line-by-line. Really? Am I misunderstanding this code that very much appears to search line by line? https://github.com/ggreer/the_silver_searcher/blob/master/src/search.c#L113-L121 https://github.com/ggreer/the_silver_searcher/blob/master/sr... GNU grep never does "throw line into buffer and then search it." GNU grep just searches the entire buffer at once.
    - ggreer 10y ago
      You didn't check to see what value opts.multiline has. It's initialized to TRUE in src/options.c. It's only set to FALSE if a user passes the undocumented --nomultiline option to ag. Default behavior with a regex query is https://github.com/ggreer/the_silver_searcher/blob/master/src/search.c#L92-L111 https://github.com/ggreer/the_silver_searcher/blob/master/sr...
      - burntsushi 10y ago
        Ah excellent, thank you for fixing my understanding. :-)

bsg75 10y ago

IIRC, Ag excels at code search, where grep can outpace it when searching things like data files. "The Silver Searcher is a tool for searching code." (http://geoff.greer.fm/ag/ http://geoff.greer.fm/ag/)