7 ms·

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or

by ggreer 10y ago

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.) 2. Not counting line numbers by default. The blog post addresses this, but I think results without line numbers are far less useful; so much so that I've traded away performance in ag. (Note that even if you tell ag not to print line numbers, it still wastes time counting them. The printing code is the result of me merging a lot of PRs that I really shouldn't have.) 3. Not using mmap(). This is a big one, and I'm not sure what the deal is here. I just added a --nommap option to ag in master.[1] It's a naive implementation, but it benchmarks comparably to the default mmap() behavior. I'm really hoping there's a flag I can pass to mmap() or madvise() that says, "Don't worry about all that synchronization stuff. I just want to read these bytes sequentially. I'm OK with undefined behavior if something else changes the file while I'm reading it." The author also points out correctness issues with ag. Ag doesn't fully support .gitiginore. It doesn't support unicode. Inverse matching (-v) can be crazy slow. These shortcomings are mostly because I originally wrote ag for myself. If I didn't use certain gitignore rules or non-ASCII encodings, I didn't write the code to support them. Some expectation management: If you try out ripgrep, don't get your hopes up. Unless you're searching some really big codebases, you won't notice the speed difference. What you will notice, however, are the feature differences. Take a look at https://github.com/BurntSushi/ripgrep/issues https://github.com/BurntSushi/ripgrep/issues to get a taste of what's missing or broken. It will be some time before all those little details are ironed-out. That said, may the best code searching tool win. :) 1. https://github.com/ggreer/the_silver_searcher/commit/bd65e2691d6ce673e026842d7e7d02bc25adfd26 https://github.com/ggreer/the_silver_searcher/commit/bd65e26...

burntsushi 10y ago
Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory maps on Windows where they seem to do a bit better. (I think my blog post gives enough details that you could reproduce the benchmark environment precisely if you were so inclined. This was important to me, so I spent a lot of time documenting it.) 3. The set of features supported by rg should be very very close to what is supported by ag. Reviewing `ag`'s man page again, probably the only things missing from rg are --ackmate, --depth, some of the color configurability flags (but rg does do coloring), --passthrough, --smart-case and --stats maybe? I might be missing some others. And Mercurial support (but ag's is incomplete). In exchange, rg gives you much better single file performance, better large-repo performance and real Unicode support that doesn't slow way down. I'd say those are pretty decent expectations. :-) Thanks for ag by the way. It and ack have definitely triggered a new kind of searching. I have some further information retrievalish ideas on evolving the concept, but those will have to wait!
- kzrdude 10y ago
  `ag --pager` is the most important one (I use alias ag='ag --pager "less -FRX"')
- ggreer 10y ago
  In terms of core features, ripgrep is totally there. It searches fast. It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy. My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of my head: Ag reads the user's global gitignore. (This is harder than most people think.) It detects stdout redirects such as "ag blah > output.txt" and ignores output.txt. It can search gz and xz files. It defaults to smart-case searching. It can limit a search to one hardware device (--one-device), avoiding slow reads on network mounts. And as a commenter already pointed out, it supports the --pager option. Taken together, all those small differences are likely to cause an average ag user some grief. I wanted to manage expectations so that users wouldn't create annoying "issues" (really, feature requests) on your GitHub repo. Sorry if that came off the wrong way. On a completely unrelated note: I see ripgrep supports .rgignore files, similar to how ag supports .agignore. It'd be nice if we could combine forces and choose a single filename for this purpose. That way when the next search tool comes along, it can use the same thing instead of .zzignore or whatever. It would also make it easier for users to switch between our tools. I'd suggest a generic name like ".ignore" or ".ignores", but I'm sure some tool creates such files or directories already. Edit: Actually, it looks like .ignore can work. The only examples I've found of .ignore files are actual files containing ignore patterns.
  - mvermaat 10y ago
    I'd say that .ignore is too generic a name. What is to be ignored by what? But I like the idea of standardizing this. Perhaps the cache directory tagging standard gives some inspiration. http://www.brynosaurus.com/cachedir/spec.html http://www.brynosaurus.com/cachedir/spec.html
  - samatman 10y ago
burntsushi 10y ago
For anyone else interested in the memory map issue, here's some more data: https://news.ycombinator.com/item?id=12567326 https://news.ycombinator.com/item?id=12567326
mintplant 10y ago
> Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support Would it be possible to detect when an expression requires PCRE-specific features and use a different engine when possible?
- ggreer 10y ago
  It's possible, but it's certainly not easy. Here are some complications: 1. The DFA-regex engine's syntax must be a subset of PCRE's syntax. If it's not, then users will be very confused when regex features work fine in isolation, but cause errors when combined in the same query. 2. The DFA-regex's behavior must be the same as PCRE. If whitespace matching or unicode support is even slightly different, it will frustrate users. 3. Adding another dependency means yet another way in which compilation can fail or incompatibilities can arise. Considering the marginal usefulness of backtracking and captures, I'd prefer to keep ag as simple as possible and ditch them.
  - deleted 10y ago
    [deleted]
Fiahil 10y ago
Thanks for bringing ag to the world. It's my favorite code searching tool because the letter 'a' and the letter 'g', from its name, feel easier to type than the ones of any other code searching tool. I am definitively willing to sacrifice a few milliseconds, while searching for a word in my megabyte of code, for this feature.
JoshTriplett 10y ago
> 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.) The standard trick here is to use the faster method for searches that it supports, and use the slower but more capable method only for searches that require it. Parse the regex, see if a DFA will work, and only use PCRE for expressions with backreferences and similar.
- raphlinus 10y ago
  Or use fancy-regex. Not ready for prime-time, but potentially the best of both worlds.
- burntsushi 10y ago
  Do you know any engines that actually do this? As in, is it really standard? I thought maybe Spencer's Tcl regex engine did it? Although I confess, I've never read the source. I guess RE2/Rust/Go all kind of do it as well. For example, RE2/Rust/Go will actually do backtracking in some cases! (It's bounded of course, maintaining linear time.) But this doesn't actually meet the criteria of being able to support more advanced features.
  - JoshTriplett 10y ago
    > Do you know any engines that actually do this? As in, is it really standard? I don't know about choosing between multiple regex engines, but GNU grep and other grep implementations check for non-regex fixed strings or trivial patterns, and have special cases for those.
    - burntsushi 10y ago
      Well... yeah... I was more thinking about really big switches like between PCRE and a DFA. It looks like the sibling comment found one that I think qualifies based on the description (assuming "NFA" is used correctly :P).
petre 10y ago
I've just installed ag. It defaults to case insensitive searches when the pattern is in lowercase. Is there a way to change this default? Perhaps a config file of some sort?
- ggreer 10y ago
  There's no config file for default options. You probably want to add an alias to your bash/zsh/fishrc: alias ag='ag -s' I tend to favor aliases over config files. It reduces complexity and improves startup time. Startup time may not seem like a big deal, but it really matters if you're running something like: find . -ctime 2 -exec ag blah (Find all files modified in the past two days and search them for instances of "blah".) If 10,000 files were changed in the past two days, and your search program takes 10 milliseconds to parse a config file, that's an extra 100 seconds wasted.
  - Scaevolus 10y ago
    find can batch arguments using +, much like xargs: find . -ctime 2 -exec ag blah {} + This makes startup latency less important.
fragmede 10y ago
> It looks like ripgrep gets most of its speedup on ag by: A non-trivial amount of time spent is simply reading the files off the disk. If speed is the all-encompassing metric, there's a big gain to be made by pre-processing the files into an index and loading that into memory instead, and that's what livegrep[1] does. If you find yourself waiting for grep often enough, it's a pretty handy internal tool to configure across all repos. [1] https://livegrep.com/about https://livegrep.com/about
- therein 10y ago
  You just said that to the author of The Silver Searcher (ag). [0] https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher

by ggreer 10y ago

burntsushi 10y ago

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory maps on Windows where they seem to do a bit better. (I think my blog post gives enough details that you could reproduce the benchmark environment precisely if you were so inclined. This was important to me, so I spent a lot of time documenting it.) 3. The set of features supported by rg should be very very close to what is supported by ag. Reviewing `ag`'s man page again, probably the only things missing from rg are --ackmate, --depth, some of the color configurability flags (but rg does do coloring), --passthrough, --smart-case and --stats maybe? I might be missing some others. And Mercurial support (but ag's is incomplete). In exchange, rg gives you much better single file performance, better large-repo performance and real Unicode support that doesn't slow way down. I'd say those are pretty decent expectations. :-) Thanks for ag by the way. It and ack have definitely triggered a new kind of searching. I have some further information retrievalish ideas on evolving the concept, but those will have to wait!

kzrdude 10y ago
`ag --pager` is the most important one (I use alias ag='ag --pager "less -FRX"')
ggreer 10y ago
In terms of core features, ripgrep is totally there. It searches fast. It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy. My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of my head: Ag reads the user's global gitignore. (This is harder than most people think.) It detects stdout redirects such as "ag blah > output.txt" and ignores output.txt. It can search gz and xz files. It defaults to smart-case searching. It can limit a search to one hardware device (--one-device), avoiding slow reads on network mounts. And as a commenter already pointed out, it supports the --pager option. Taken together, all those small differences are likely to cause an average ag user some grief. I wanted to manage expectations so that users wouldn't create annoying "issues" (really, feature requests) on your GitHub repo. Sorry if that came off the wrong way. On a completely unrelated note: I see ripgrep supports .rgignore files, similar to how ag supports .agignore. It'd be nice if we could combine forces and choose a single filename for this purpose. That way when the next search tool comes along, it can use the same thing instead of .zzignore or whatever. It would also make it easier for users to switch between our tools. I'd suggest a generic name like ".ignore" or ".ignores", but I'm sure some tool creates such files or directories already. Edit: Actually, it looks like .ignore can work. The only examples I've found of .ignore files are actual files containing ignore patterns.
- mvermaat 10y ago
  I'd say that .ignore is too generic a name. What is to be ignored by what? But I like the idea of standardizing this. Perhaps the cache directory tagging standard gives some inspiration. http://www.brynosaurus.com/cachedir/spec.html http://www.brynosaurus.com/cachedir/spec.html
- samatman 10y ago

burntsushi 10y ago

For anyone else interested in the memory map issue, here's some more data: https://news.ycombinator.com/item?id=12567326 https://news.ycombinator.com/item?id=12567326

mintplant 10y ago

> Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support Would it be possible to detect when an expression requires PCRE-specific features and use a different engine when possible?

ggreer 10y ago
It's possible, but it's certainly not easy. Here are some complications: 1. The DFA-regex engine's syntax must be a subset of PCRE's syntax. If it's not, then users will be very confused when regex features work fine in isolation, but cause errors when combined in the same query. 2. The DFA-regex's behavior must be the same as PCRE. If whitespace matching or unicode support is even slightly different, it will frustrate users. 3. Adding another dependency means yet another way in which compilation can fail or incompatibilities can arise. Considering the marginal usefulness of backtracking and captures, I'd prefer to keep ag as simple as possible and ditch them.
- deleted 10y ago
  [deleted]

Fiahil 10y ago

Thanks for bringing ag to the world. It's my favorite code searching tool because the letter 'a' and the letter 'g', from its name, feel easier to type than the ones of any other code searching tool. I am definitively willing to sacrifice a few milliseconds, while searching for a word in my megabyte of code, for this feature.

JoshTriplett 10y ago

> 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.) The standard trick here is to use the faster method for searches that it supports, and use the slower but more capable method only for searches that require it. Parse the regex, see if a DFA will work, and only use PCRE for expressions with backreferences and similar.

raphlinus 10y ago
Or use fancy-regex. Not ready for prime-time, but potentially the best of both worlds.
burntsushi 10y ago
Do you know any engines that actually do this? As in, is it really standard? I thought maybe Spencer's Tcl regex engine did it? Although I confess, I've never read the source. I guess RE2/Rust/Go all kind of do it as well. For example, RE2/Rust/Go will actually do backtracking in some cases! (It's bounded of course, maintaining linear time.) But this doesn't actually meet the criteria of being able to support more advanced features.
- JoshTriplett 10y ago
  > Do you know any engines that actually do this? As in, is it really standard? I don't know about choosing between multiple regex engines, but GNU grep and other grep implementations check for non-regex fixed strings or trivial patterns, and have special cases for those.
  - burntsushi 10y ago
    Well... yeah... I was more thinking about really big switches like between PCRE and a DFA. It looks like the sibling comment found one that I think qualifies based on the description (assuming "NFA" is used correctly :P).

petre 10y ago

I've just installed ag. It defaults to case insensitive searches when the pattern is in lowercase. Is there a way to change this default? Perhaps a config file of some sort?

ggreer 10y ago
There's no config file for default options. You probably want to add an alias to your bash/zsh/fishrc: alias ag='ag -s' I tend to favor aliases over config files. It reduces complexity and improves startup time. Startup time may not seem like a big deal, but it really matters if you're running something like: find . -ctime 2 -exec ag blah (Find all files modified in the past two days and search them for instances of "blah".) If 10,000 files were changed in the past two days, and your search program takes 10 milliseconds to parse a config file, that's an extra 100 seconds wasted.
- Scaevolus 10y ago
  find can batch arguments using +, much like xargs: find . -ctime 2 -exec ag blah {} + This makes startup latency less important.

fragmede 10y ago

> It looks like ripgrep gets most of its speedup on ag by: A non-trivial amount of time spent is simply reading the files off the disk. If speed is the all-encompassing metric, there's a big gain to be made by pre-processing the files into an index and loading that into memory instead, and that's what livegrep[1] does. If you find yourself waiting for grep often enough, it's a pretty handy internal tool to configure across all repos. [1] https://livegrep.com/about https://livegrep.com/about

therein 10y ago
You just said that to the author of The Silver Searcher (ag). [0] https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher