4 ms·

GNU grep is fast, but there are certain alternatives that are pretty fast (probably faster): - ack [0] - ag (also known as the silver searcher) [1] You co

by 0xmohit 10y ago

GNU grep is fast, but there are certain alternatives that are pretty fast (probably faster): - ack [0] - ag (also known as the silver searcher) [1] You could make `grep` faster by telling it to interpret the pattern as a fixed string (-F or --fixed-strings) is you're not using a regex. Setting to LANG to C would also probably make it faster. [0] http://beyondgrep.com/ http://beyondgrep.com/ [1] https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher

burntsushi 10y ago
I think it would be interesting to compare those utilities with GNU grep on a single file, because I understand them to be faster on many files because they (1) search files in parallel and (2) skip many uninteresting files automatically. I also don't think either of those tools are DFA based like GNU grep. One thing I'd like to see is combine many of the performance tricks found in GNU grep with the aforementioned (1)+(2) techniques. > You could make `grep` faster by telling it to interpret the pattern as a fixed string (-F or --fixed-strings) is you're not using a regex. That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. If it's a literal, it should be able to stay out of the main DFA engine entirely.
- gpderetta 10y ago
  I use grep for single files (mostly because it is burned in my muscle memory). The silver searcher (from emacs) for bulk searches in a large directory tree (like a codebase).
- nilved 10y ago
  It's pretty easy to use grep and search in parallel while skipping uninteresting files. `git ls-files | xargs -P4 -n8 grep whatever` is a good way to do this; or `git grep`, but that's using PCRE and not parallelized. > That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. If it's a literal, it should be able to stay out of the main DFA engine entirely. That would be clever but it doesn't do that.
  - goldenkey 10y ago
    The regex is compiled anyhow to a C-call subroutine using a lib. So the overhead is only a function call. I highly doubt a single inlined proc is going to give much speed gain.
  - burntsushi 10y ago
    > That would be clever but it doesn't do that. Really? Pretty sure it does. Have you read the source code? There is a specific part that looks to see if it has a literal and will use that. If it has an exact match, then it will skip the DFA completely.
- 0xmohit 10y ago
  > That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. It does help when the search string comprises of regex metacharacters and you actually wanted a literal search.
michaelmior 10y ago
While ag is fast, I use it because of added features. Specifically the ability to automatically ignore files which are ignored in your project's VCS.
- agentgt 10y ago
  It is also because it is multithreaded (IIRC). There is another one written in Go that is for sure. GNU grep is faster one file (at least the last time I tested).
  - asamarin 10y ago
    You're probably talking about sift https://sift-tool.org/ https://sift-tool.org/
coldtea 10y ago
I've found pt (the platinum searcher) to be better (and even faster) than ag for codebase searches (and it can automatically pick up .gitignore etc). https://github.com/monochromegane/the_platinum_searcher https://github.com/monochromegane/the_platinum_searcher
- foobar__ 10y ago
  How is it better?
  - coldtea 10y ago
    From their page: It searches code about 3–5× faster than ack. It searches code as fast as the_silver_searcher(ag). It ignores file patterns from your .gitignore. It searches UTF-8, EUC-JP and Shift_JIS files. It provides binaries for multi platform (Mac OS X, Windows, Linux). But even more so, sensible defaults that work nicely for code search use cases. And written in Go, without accumulated cruft for 200 platforms and POSIX subtleties like grep, so even someone like me can hack it to do something special if I want to.
    - epmatsw 10y ago
      Ag handles .gitignore as well. And has defaults appropriate for code searching. So the difference is official Windows support?
      - coldtea 10y ago
        It's also faster (30% to 50% for my codebase), written in Go, and easier to extend. $ time ag zmq | wc -l 63 real 0m0.017s user 0m0.022s sys 0m0.016s $ time pt zmq | wc -l 296 real 0m0.013s user 0m0.014s sys 0m0.017s (Timing differences consistent over multiple runs of both on the same codebase -- no disc cache effect) -- the wc difference is because of different "surrounding context" setting.
        10y ago

by 0xmohit 10y ago

burntsushi 10y ago

I think it would be interesting to compare those utilities with GNU grep on a single file, because I understand them to be faster on many files because they (1) search files in parallel and (2) skip many uninteresting files automatically. I also don't think either of those tools are DFA based like GNU grep. One thing I'd like to see is combine many of the performance tricks found in GNU grep with the aforementioned (1)+(2) techniques. > You could make `grep` faster by telling it to interpret the pattern as a fixed string (-F or --fixed-strings) is you're not using a regex. That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. If it's a literal, it should be able to stay out of the main DFA engine entirely.

gpderetta 10y ago
I use grep for single files (mostly because it is burned in my muscle memory). The silver searcher (from emacs) for bulk searches in a large directory tree (like a codebase).
nilved 10y ago
It's pretty easy to use grep and search in parallel while skipping uninteresting files. `git ls-files | xargs -P4 -n8 grep whatever` is a good way to do this; or `git grep`, but that's using PCRE and not parallelized. > That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. If it's a literal, it should be able to stay out of the main DFA engine entirely. That would be clever but it doesn't do that.
- goldenkey 10y ago
  The regex is compiled anyhow to a C-call subroutine using a lib. So the overhead is only a function call. I highly doubt a single inlined proc is going to give much speed gain.
- burntsushi 10y ago
  > That would be clever but it doesn't do that. Really? Pretty sure it does. Have you read the source code? There is a specific part that looks to see if it has a literal and will use that. If it has an exact match, then it will skip the DFA completely.
0xmohit 10y ago
> That doesn't really make much sense. GNU grep can tell whether a pattern is just a literal or not. It does help when the search string comprises of regex metacharacters and you actually wanted a literal search.

michaelmior 10y ago

While ag is fast, I use it because of added features. Specifically the ability to automatically ignore files which are ignored in your project's VCS.

agentgt 10y ago
It is also because it is multithreaded (IIRC). There is another one written in Go that is for sure. GNU grep is faster one file (at least the last time I tested).
- asamarin 10y ago
  You're probably talking about sift https://sift-tool.org/ https://sift-tool.org/

coldtea 10y ago

I've found pt (the platinum searcher) to be better (and even faster) than ag for codebase searches (and it can automatically pick up .gitignore etc). https://github.com/monochromegane/the_platinum_searcher https://github.com/monochromegane/the_platinum_searcher

foobar__ 10y ago
How is it better?
- coldtea 10y ago
  From their page: It searches code about 3–5× faster than ack. It searches code as fast as the_silver_searcher(ag). It ignores file patterns from your .gitignore. It searches UTF-8, EUC-JP and Shift_JIS files. It provides binaries for multi platform (Mac OS X, Windows, Linux). But even more so, sensible defaults that work nicely for code search use cases. And written in Go, without accumulated cruft for 200 platforms and POSIX subtleties like grep, so even someone like me can hack it to do something special if I want to.
  - epmatsw 10y ago
    Ag handles .gitignore as well. And has defaults appropriate for code searching. So the difference is official Windows support?
    - coldtea 10y ago
      It's also faster (30% to 50% for my codebase), written in Go, and easier to extend. $ time ag zmq | wc -l 63 real 0m0.017s user 0m0.022s sys 0m0.016s $ time pt zmq | wc -l 296 real 0m0.013s user 0m0.014s sys 0m0.017s (Timing differences consistent over multiple runs of both on the same codebase -- no disc cache effect) -- the wc difference is because of different "surrounding context" setting.
      - 10y ago