6 ms·
It's a simple break down of communication. I try to be upfront about this, but no matter how hard I try or how many times I try to clarify it, I can't stop the
by burntsushi 8y ago
It's a simple break down of communication. I try to be upfront about this, but
no matter how hard I try or how many times I try to clarify it, I can't stop
the spread of inaccurate claims. (And my goodness, this entire HN thread has
numerous inaccuracies just with GNU grep alone.)
On equivalent tasks, ripgrep is not orders of magnitude faster than GNU grep,
outside of pathological cases that involve Unicode support. (I can provide
evidence for that if you like.)
For example, in my checkout of the Linux kernel, here's a recursive grep that
searches everything:
$ time LC_ALL=C grep -ar PM_RESUME | wc -l
17
real 1.176
user 0.758
sys 0.407
maxmem 7 MB
faults 0
Now compare that with ripgrep, with a command that uses the same amount of
parallelism and searches the same amount of data:
$ time rg -j1 -uuu PM_RESUME | wc -l
17
real 0.581
user 0.187
sys 0.384
maxmem 7 MB
faults 0
Which is 2x faster, but not "order of magnitude." Now compare it with how long
ripgrep takes using the default command:
$ time rg PM_RESUME | wc -l
17
real 0.125
user 0.646
sys 0.654
maxmem 19 MB
faults 0
At 10x faster, this is where you start to get to "order of magnitude" faster
claims. But for someone who cares about precise claims with respect to
performance, this is uninteresting because ripgrep is 1) using parallelism and
2) skipping some files due to `.gitignore` and other such rules.
You can imagine that if your directory has a lot of large binary files, or if
you're searching in a directory with high latency (a network mount), then you
might see even bigger differences from ripgrep without generally seeing a
difference in search results because ripgrep tends to skip things you don't
care about anyway.
In summary, there is an impedance mismatch when talking about performance
because most people don't have a good working mental model of how these tools
work internally. Many people report on their own perceived performance
improvements and compare that directly to how they used to use grep. They
aren't wrong in a certain light, because ultimately, the user experience is
what matters. But of course, they are wrong in another light if you're
interpreting it as a precise technical claim about the performance
characteristics of a program.
- tlamponi 8y ago> [...] because ripgrep tends to skip things you don't care about anyway. so you bring the trick #1 from the grep author to a new level: > #1 trick: GNU grep is fast because it AVOIDS LOOKING AT EVERY INPUT BYTE. You stop looking at entire files, which can avoid a lot of bytes to go through. Also parallelisms surely helps with a lot of files, or big ones really big ones which could be processed in chunks.
- burntsushi 8y agoNo, that's a bad trick to rely on for speed these days. GNU grep's speed comes from using memchr in its skip loop. Skipping bytes is only going to matter when the thing you're searching for is longer. For short needles, there isn't much opportunity for skipping, so it's better to say inside a vectorized routine like memchr (which, for glibc, is written in Assembly on most platforms).
- urmish 8y agoExcellent tool. rg is very fast. I am using it at work now :)
- TheGrassyKnoll 8y agoSecond that. Well done.
- elcomet 8y agoSo if you cripple ripgrep, to act like grep, it is not much faster. This is kind of obvious. Ripgrep is faster because it makes assumptions, that will hold true for a large part of its users. This is like saying "if I remove wings from a plane, then it won't go much faster than my car. So a plane is not technically faster than a car" Of course, users should know the difference between grep and ripgrep (especially with tweaks like .gitignore, which could be confusing if you don't know).
- hvis 8y agoThey should also know that git ls-files -z | xargs -0 grep -nHE <regexp> can be almost as fast.
- dralley 8y agoWhoa, which "time" command are you using that provides memory and page fault info?
- Retra 8y agoFrom what I can tell, the GNU time command has a -f "%F" flag that will provide it. http://man7.org/linux/man-pages/man1/time.1.html http://man7.org/linux/man-pages/man1/time.1.html
- burntsushi 8y agoI'm using the `time` built-in from zsh, with this config in my ~/.zshrc: TIMEFMT=$'\nreal\t%*E\nuser\t%*U\nsys\t%*S\nmaxmem\t%M MB\nfaults\t%F'
- bravura 8y agoFYI one thing I have found that rg is much slower than GNU grep is when I have 100K exact match patterns.
- burntsushi 8y agoYup, that's a known bug. It will be fixed in the next release. I've already done the leg work to fix it, and it should be on master soon. My local copy of ripgrep can now search with my local /usr/share/dict/words as exact patterns just as fast as grep.
- eadmund 8y ago> Which is 2x faster, but not "order of magnitude." Well, it's a base-2 order of magnitude!
- jonstewart 8y ago^ this. For common usage of searching for decently long patterns (>3-6 bytes) with several literal characters/limited wildcarding on a text with a largish alphabet, it is algorithmically very difficult to do _far_ better than GNU Grep’s Boyer-Moore-based implementation. That situation changes when you have very short patterns, very long patterns, many patterns, or small alphabets (eg DNA). As burnsushi notes, ripgrep’s performance difference for common feel usage comes more from being smarter about its input than from algorithms (its algorithms are solid, of course).