18 ms·

Ripgrep – A new command line search tool

740 points by anp 10y ago

xuhu 10y ago
Why not make --with-filename default even for e.g. "rg somestring" ? That seems like it could hinder adoption since grep does it and it's useful when piping to other commands. Is it enabled when you specify a directory (rg somestring .) ?
- burntsushi 10y ago
  It should be the default whenever you search more than one file.
fsiefken 10y ago
nice, but does it compile and run on armhf? I don't see any binaries
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- burntsushi 10y ago
  I'm not terribly familiar with Rust's story on ARM, but I do know there are people working on cross compilation. I'll see if I can hook it up to the CI later today. Certainly, there's nothing in ripgrep itself that should prevent it from working on ARM.
  - echelon 10y ago
    I cross compile my home automation sever that I wrote in Rust to ARM. It should work.
    - Cshelton 10y ago
      I'll be doing that soon! Care to share your code? or explain what exactly you're doing with it?
      - echelon 10y ago
        I've published a few crates in the home automation space, but they're both pretty terrible at this point. One of them is for controlling WeMo devices, and now that I know Rust better I'm going to rewrite it using the new blocking APIs rather than Mio. (And real SOAP support.) What kind of home automation will you be doing? I'd absolutely love to collaborate on a gateway that interacts with Wemo, Z-wave, Zigbee, etc. and can do basic things like events, scenes, schedules, etc. I haven't focused on home automation all summer long since I've been trying to get my Donald Trump TTS engine (also written in Rust) ready for the election. It's soooo close.
- mbrubeck 10y ago
  I was able to build ripgrep from source for ARM, cross-compiling from my laptop running Debian, following the directions here: https://github.com/japaric/rust-cross#tldr-ubuntu-example https://github.com/japaric/rust-cross#tldr-ubuntu-example (I haven't actually run it because I don't have an ARM linux device handy.)
  - pyroholic 10y ago
    Cross-compiled and ran a couple basic searches on an armv7l device. So at least the basic functionality works just fine.
dikaiosune 10y ago
Compiling it to try right now... Some discussion over on /r/rust: https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_faster_than_grep_ag_git_grep_ucg_pt/ https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_fas... EDIT: The machine I'm on is much less beefy than the benchmark machines, which means that the speed difference is quite noticeable for me.
echelon 10y ago
Rust is really staring to be seen in the wild now.
- CJefferson 10y ago
  I agree, and I'm already using both ripgrep and rust-parallel ( https://github.com/mmstick/parallel https://github.com/mmstick/parallel , a gnu-parallel replacement which should probably get another name ). I am really happy to see Rust apps actually been written -- Rust programmers seem to be actually trying to replace the existing code lying around, instead of just insulting it and telling us how much better the world would be if we used their language.
  - ekidd 10y ago
    Honestly, for a certain kind of developer—I haven't fingerpointed exactly which kind—writing Rust code is dangerously addictive. There's some combination of aggressive type checking, tooling, and expressiveness that just hits a sweet spot in my brain. Rust fills roughly the same niche in my brain as old-school C++, except it also lets me do some rudimentary functional stuff, there are virtually no footguns, and the CLI tooling is out of this world. And I've long since made friends with the borrow checker. So it's tempting to just spend my life cranking out Rust code as fast as I can. It's actually a problem. :-)
    - CJefferson 10y ago
      Well in the interests of making contraversal statements, that's now two Rust programs I plan on using day-to-day, and zero for Haskell, Scala, and I'm sure a few other languages I'm not remembering.
      - unfamiliar 10y ago
        It will only be a matter of time before someone rewrites xmonad in Rust.
        10y ago
pixelbeat 10y ago
Thanks for the detailed comparisons and writeup. I find this simple wrapper around grep(1) very fast and useful: http://www.pixelbeat.org/scripts/findrepo http://www.pixelbeat.org/scripts/findrepo
- burntsushi 10y ago
  Thanks for the kind words! Note that the key thing that `findrepo` doesn't support is respecting your .gitignore files. For example, in the Rust ecosystem, we often have a `target` directory in our projects that contains a lot of stuff we probably don't want to search. In fact, running `cargo new` will add that directory to your `.gitignore` automatically! Tools like The Silver Searcher and ripgrep will ignore that directory (and all others like it) automatically. There are other advantages to ripgrep. For example, every other tool that supports Unicode as well as ripgrep (that's `git grep` and GNU grep) experience a substantial slow down when trying to use more advanced Unicode features (like \w, -i, etc.). This is one of the things my benchmarks show.
  - pixelbeat 10y ago
    Yes ignoring .gitignore contents is a very useful feature. That could be added quite easily to findrepo though would probably have to be an opt as it is often useful to search intermediate build files etc. that aren't checked in. Whereas `git grep` handles the other use case of only searching checked in files. Interesting info wrt efficient unicode processing for \w and -i. cheers
cm3 10y ago
To build a static Linux binary with SIMD support, run this: RUSTFLAGS="-C target-cpu=native" rustup run nightly cargo build --target x86_64-unknown-linux-musl --release --features simd-accel
- burntsushi 10y ago
  That's an awesome demonstration of how easy it is to swap out the libc in Rust. :-) Note that I also distribute statically compiled executables with musl and SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
  - cm3 10y ago
    > That's an awesome demonstration of how easy it is to swap out the libc in Rust. :-) Now I have to look up how I use cargo to build a static binary on FreeBSD, where I don't have to swap out libc. > Note that I also distribute statically compiled executables with musl and SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases I took the flag from your blog post, thanks for pointing out the explicit feature flag. That will allow the binary to run on more cpus.
jonstewart 10y ago
Nice! Lightgrep[1] uses libicu et al to look up code points for a user-specified encoding and encode them as bytes, then just searches for the bytes. Since ripgrep is presumably looking just for bytes, too, and compiling UTF-8 multibyte code points to a sequence of bytes, perhaps you can do likewise with ICU and support other encodings. ICU is a bear to build against when cross-compiling, but it knows hundreds of encodings, all of the proper code point names, character classes, named properties, etc., and the surface area of its API that's required for such usage is still pretty small. [1]: http://strozfriedberg.github.io/liblightgrep http://strozfriedberg.github.io/liblightgrep
- burntsushi 10y ago
  I hadn't heard of liblightgrep, nice. It's on my short list for looking more closely. I doubt I'd ever be comfortable with Rust's regex engine growing a dependency on libicu, but it's still worth understanding your implementation. Some questions, if you don't mind. The big one is: does your regex engine use finite automata, and does it put the text decoding into the automaton itself? For example, when you compile the `.` regex, you end up with an automaton that inlines UTF-8 decoding itself. It looks like this: https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fabbd0a3 https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fa... Does you regex library do that for each type of encoding? Or is there a transcoding step?
  - jonstewart 10y ago
    Reply fail, see above.
- jonstewart 10y ago
  Yep! It's a multipattern engine so we use an NFA--slower than a normal engine, but effective when you have thousands of patterns to search for in binary streams. Since we are often searching binary, we'll encounter text fragments in lots of different encodings, so they're all compiled into the NFA. We wrote a paper about its Unicode support: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper....
qwertyuiop924 10y ago
That is really cool. Although I think this is a case where Good Enough will beat amazing, at least for me (especially given how much I use backrefs).
spicyj 10y ago
rg is harder to type with one hand because it uses the same finger twice. :)
- pixelbath 10y ago
  But 'grep' doesn't?
  - echelon 10y ago
    True, but `ag` and `ack` don't. This is a really interesting point. If we're going to be typing commands frequently, it would make sense for them to be optimally efficient to type using QWERTY (and maybe other common keyboard layouts). `ls` and `cd` are super easy to type.
- adelarsq 10y ago
  lol
- robinson-wall 10y ago
  Off topic, but you've just made me realise I don't always use the same finger when I type the letter R. If I were typing lower case "rg" I use middle finger for r and then index finger for g. I'm now thinking about it too much to be able to type naturally, but I seem to use middle and index interchangeably for some keys depending on which key was just pressed / is next to be pressed. ... and I just noticed if I sometimes use different fingers from the "wrong" hand to type depending on context too! I've typed the letter Y using right-index, left-index and left-middle and left-index during the course of writing this post. I guess this is what learning to touch type without being actually taught buys you :)
  - auganov 10y ago
    Was just about to write a similar reply too. I type "rg" the same way. My left hand very often goes for the YHN column, the right hand won't ever touch TGB though. For typing "lol" I'll very often just move my right hand to the right (or sort of angle it) and press the l with the index and the o with the middle finger. I've been plateaued at a 110wpm avg (121 top speed) @ 10fastfingers for quite a few months now, so perhaps breaking some of these bad habits would help me get to the desired 140.
chalana 10y ago
I'm never sure whether or not I should adopt these fancy new command line tools that come out. I get them on my muscle memory and then all of a sudden I ssh into a machine that doesn't have any of these and I'm screwed...
- petdance 10y ago
  I have to switch back and forth between ack and grep all the time. Sometimes I use ack, sometimes I use grep. I wrote ack, but I've never stopped using grep, and ack has never been intended as a replacement for grep. For most of the common ack flags, they are the same as grep. -i, -l, -v, -w, -A, -B, -C, etc. That was intentional, to minimize that whiplash. One other suggestion is that you drop ack into your ~/bin directory that you sync between machines. It's a single Perl file, and that portability is a feature. As long as your machine has a Perl on it, you can use ack.
- sitkack 10y ago
  Just make your ssh script install all the binaries where ever you go! ssh-caravan
losvedir 10y ago
Meh, yet another grep tool.... wait, by burntsushi! Whenever I hear of someone wanting to improve grep I think of the classic ridiculous fish piece[0]. But when I saw that this one was by the author of rust's regex tools, which I know from a previous post on here, are quite sophisticated, I perked up. Also, the tool aside, this blog post should be held up as the gold standard of what gets posted to hacker news: detailed, technical, interesting. Thanks for your hard work! Looking forward to taking this for a spin. [0] http://ridiculousfish.com/blog/posts/old-age-and-treachery.html http://ridiculousfish.com/blog/posts/old-age-and-treachery.h...
- trishume 10y ago
  Totally agree. This is like burntsushi's technical tour de force, the culmination of years of work building up a stack of technology leading up to this utility and article. It's a simple tool, but he's built every piece from the ground up to be the very best, and then documented the shit out of it. So cool.
- pohl 10y ago
  Another burntsushi project was recently posted, but didn't get much attention: https://news.ycombinator.com/item?id=12559515 https://news.ycombinator.com/item?id=12559515
  - skanga 10y ago
    Looks interesting. But I did not find binaries for it and do not want to setup a rust env to try it out.
    - luckman212 10y ago
      The releases are right there on GitHub: https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
      - burntsushi 10y ago
        I think the GP was asking about xsv, not ripgrep. There are binary releases for xsv though: https://github.com/BurntSushi/xsv/releases https://github.com/BurntSushi/xsv/releases
bodyfour 10y ago
It would be interesting to benchmark how much mmap hurts when operating in a non-parallel mode. I think a lot of the residual love for mmap is because it actually did give decent results back when single core machines were the norm. However, once your program becomes multithreaded it imposes a lot of hidden synchronization costs, especially on munmap(). The fastest option might well be to use mmap sometimes but have a collection of single-thread processes instead of a single multi-threaded one so that their VM maps aren't shared. However, this significantly complicates the work-sharing and output-merging stages. If you want to keep all the benefits you'd need a shared-memory area and do manual allocation inside it for all common data which would be a lot of work. It might also be that mmap is a loss these days even for single-threaded... I don't know. Side note: when I last looked at this problem (on Solaris, 20ish years ago) one trick I used when mmap'ing was to skip the "madvise(MADV_SEQUENTIAL)" if the file size was below some threshold. If the file was small enough to be completely be prefetched from disk it had no effect and was just a wasted syscall. On larger files it seemed to help, though.
- karmakaze 10y ago
  I recently questioned if/why parallel mmap might be slower without a satisfactory conclusion. One specific thing I couldn't answer is if reusing the same filesystem buffer and program memory addresses has a less negative effect than reading a wide range of mapped memory addresses.
  - brandmeyer 10y ago
    Since all processors must share the mapping, - The initial mapping of each file in any thread must halt all of the threads which are otherwise active. - Every page fault in any mapping must also halt all of the threads. Worse, since the page tables are getting munged, some or all of the TLB cache is getting flushed every time, again, on every processor. I'm not sure of the details, but this hypothesis should be directly testable. IIRC, there are some hardware performance counters for time spent waiting on TLB lookups. Addendum: One other possibility is that the mere act of extending the working set size (of the address space) is blowing the TLB cache.
petre 10y ago
Does it use PCRE (not the lib, the regex style). If not, ack is just fine. My main concern with grep are Posix regular expressions.
- steveklabnik 10y ago
  "regex style" is a bit too broad. The regexes feel more like PCRE syntactically than they do POSIX, but does not support the more exotic features of PCRE.
- Occivink 10y ago
  It uses the re2 style, as provided by the rust regex library.
Tim61 10y ago
I love the layout of this article. Especially the pitch and anti-pitch. I wish more more tools/libraries/things would make note of their downsides. I'm convinced to give it a try.
minimax 10y ago
In contrast, GNU grep uses libc’s memchr, which is standard C code with no explicit use of SIMD instructions. However, that C code will be autovectorized to use xmm registers and SIMD instructions, which are half the size of ymm registers. I don't think this is correct. glibc has architecture specific hand rolled (or unrolled if you will lol) assembly for x64 memchr. See here: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/memrchr.S;h=840de30cd71ba96b3ae43540e6ac255c28906cc5;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86...
- burntsushi 10y ago
  Drats, you're totally right. It's easy to mess up that kind of thing. Thankfully, it looks like my analysis remains mostly unchanged. I don't see any AVX2 in there (and indeed, I didn't when I looked at the profile either, in contrast to Go's implementation). I updated the blog, thanks again for the clarification.
libman 10y ago
Tragically the news that LLVM is switching to a non-Copyfree license (see copyfree.org/standard/rejected) has ruined everything... Nothing written in Rust can be called Free Software. :(
- alphapapa 10y ago
  According to that site, Free Software (as defined by the FSF) is not Copyfree. As far as I can tell, Copyfree is like the BSD license, except you can only restrict changes to the license text itself; the license itself may not be changed or added to, but everything else in the project may be used and changed in any way. So, effectively, it seems like a license where the license itself is copyrighted with all rights reserved, but the rest of the project is public domain. So, just like BSD, anyone can take a Copyfree project proprietary, turning it non-free. The GPL is as important today as ever.
krylon 10y ago
When I use grep (which is fairly regularly), the bottleneck is nearly always the disk or the network (in case of NFS/SMB volumes). Just out of curiosity, what kind of use case makes grep and prospective replacements scream? The most "hardcore" I got with grep was digging through a few gigabytes of ShamePoint logs looking for those correlation IDs, and that apparently was completely I/O-bound, the CPUs on that machine stayed nearly idle.
- nolemurs 10y ago
  `grep` itself is admirably fast on single files, and if you're just searching a small number of files, the bottleneck will be IO, and `grep` will be about as fast as anything. But if you want to use `grep` recursively on a selected subset of millions of files it suffers a bit. `grep` assumes that if you want to collect some subset of files to search, you'll do that with some other utility, and pass the list of files to `grep`, but that creates a bottleneck. More recent search programs like `ack` and `ag` solve the multifile bottleneck issue, but also perform worse than `grep` on single large files.
  - krylon 10y ago
    Thank you!
  - burntsushi 10y ago
    > and `grep` will be about as fast as anything I'd encourage you to check out the benchmarks in my blog, especially the subtitle benchmarks, because this isn't actually true in the case of ripgrep. :-)
    - nolemurs 10y ago
      I take it back! I'm definitely switching over to using ripgrep for the next bit. Off topic - is there a way to exclude directories? I tried doing things like `rg pattern -g '!my_dir/' but nothing I tried seemed to work.
      - 10y ago
pmontra 10y ago
It looks very good and I'd like to try it. However I'm lazy and I don't want to install all the Rust dev environment to compile it. Did anybody build a .deb for Ubuntu 16?
- steveklabnik 10y ago
  If you don't want to compile it yourself, the blog post has links to binaries. Since they're entirely self-contained, you don't need a full .deb to compile them; just delete the binary when you want to get rid of it.
  - pmontra 10y ago
    I didn't notice that. Thanks. It works. The binary is 11 MB, or 1.5 MB stripped. ag is 69176 bytes. Shared libraries were a good invention :-) Sooner or later rg will be packaged properly too.
    - steveklabnik 10y ago
      Yes, especially given that rustc/cargo are appearing in distro's package sets already. Exciting times :)
    - cgag 10y ago
      I'd much rather have a self contained binary than 11 more MB of free space.
serge2k 10y ago
> We will attempt to do the impossible Oh well. Waste of time then.
- bo1024 10y ago
  Maybe the author hasn't had breakfast yet and has only done 5 impossible things so far?
zatkin 10y ago
>It is not, strictly speaking, an interface compatible “drop-in” replacement for both, but the feature sets are far more similar than different.
lobster_johnson 10y ago
Very nice. Not only fast, but feels modern. Tried it out on a 3.5GB JSON file: # rg rg erzg4 k.json > /dev/null 1.80s user 2.54s system 53% cpu 8.053 total # rg with 4 threads rg -j4 erzg4 k.json > /dev/null 1.76s user 1.29s system 99% cpu 3.059 total # OS X grep grep erzg4 k.json > /dev/null 60.62s user 0.96s system 99% cpu 1:01.75 total # GNU Grep ggrep erzg4 k.json > /dev/null 1.96s user 1.43s system 88% cpu 2.691 total GNU Grep wins, but it's pretty crusty, especially with regards to its output (even with colourization).
- burntsushi 10y ago
  My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory. This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and didn't record any measurements. I then ran them another 10 times in which I recorded them. You can see the raw output here: https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2016-09-20-ubuntu1604-ec2/raw.csv https://github.com/BurntSushi/ripgrep/blob/master/benchsuite... In any case, on my underpowered Mac, here are some results on a 1.2 GB file (notice how much the time fluctuates until its fully in cache): mac:~ andrew$ ggrep --version ggrep (GNU grep) 2.25 Packaged by Homebrew mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m5.447s user 0m0.600s sys 0m0.350s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m1.247s user 0m0.549s sys 0m0.264s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.803s user 0m0.542s sys 0m0.259s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.805s user 0m0.544s sys 0m0.260s And now for rg: mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m1.062s user 0m0.339s sys 0m0.333s mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.640s user 0m0.337s sys 0m0.302s mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.637s user 0m0.336s sys 0m0.300s Oh! And check this out, on a Mac, not using a memory map for single files is faster. My goodness---memory map performance is all over the place. mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl --no-mmap > /dev/null real 0m0.445s user 0m0.170s sys 0m0.274s If I do this on my Linux machine on the same file, I get timings of 0.275s for rg, 0.398s for rg with no memory maps (opposite direction for Mac) and 0.708s for GNU grep (v 2.25). Benchmarks are fun, eh?
ggreer 10y ago
I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.) 2. Not counting line numbers by default. The blog post addresses this, but I think results without line numbers are far less useful; so much so that I've traded away performance in ag. (Note that even if you tell ag not to print line numbers, it still wastes time counting them. The printing code is the result of me merging a lot of PRs that I really shouldn't have.) 3. Not using mmap(). This is a big one, and I'm not sure what the deal is here. I just added a --nommap option to ag in master.[1] It's a naive implementation, but it benchmarks comparably to the default mmap() behavior. I'm really hoping there's a flag I can pass to mmap() or madvise() that says, "Don't worry about all that synchronization stuff. I just want to read these bytes sequentially. I'm OK with undefined behavior if something else changes the file while I'm reading it." The author also points out correctness issues with ag. Ag doesn't fully support .gitiginore. It doesn't support unicode. Inverse matching (-v) can be crazy slow. These shortcomings are mostly because I originally wrote ag for myself. If I didn't use certain gitignore rules or non-ASCII encodings, I didn't write the code to support them. Some expectation management: If you try out ripgrep, don't get your hopes up. Unless you're searching some really big codebases, you won't notice the speed difference. What you will notice, however, are the feature differences. Take a look at https://github.com/BurntSushi/ripgrep/issues https://github.com/BurntSushi/ripgrep/issues to get a taste of what's missing or broken. It will be some time before all those little details are ironed-out. That said, may the best code searching tool win. :) 1. https://github.com/ggreer/the_silver_searcher/commit/bd65e2691d6ce673e026842d7e7d02bc25adfd26 https://github.com/ggreer/the_silver_searcher/commit/bd65e26...
- burntsushi 10y ago
  Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory maps on Windows where they seem to do a bit better. (I think my blog post gives enough details that you could reproduce the benchmark environment precisely if you were so inclined. This was important to me, so I spent a lot of time documenting it.) 3. The set of features supported by rg should be very very close to what is supported by ag. Reviewing `ag`'s man page again, probably the only things missing from rg are --ackmate, --depth, some of the color configurability flags (but rg does do coloring), --passthrough, --smart-case and --stats maybe? I might be missing some others. And Mercurial support (but ag's is incomplete). In exchange, rg gives you much better single file performance, better large-repo performance and real Unicode support that doesn't slow way down. I'd say those are pretty decent expectations. :-) Thanks for ag by the way. It and ack have definitely triggered a new kind of searching. I have some further information retrievalish ideas on evolving the concept, but those will have to wait!
hxn 10y ago
Looks like every tool has its upsides and downsides. This one lacks full PCRE syntax support. Does one have to install Rust to use it?
- kod 10y ago
  No. binaries: http://blog.burntsushi.net/ripgrep/#installation http://blog.burntsushi.net/ripgrep/#installation
- heinrich5991 10y ago
  No, Rust tools don't need a Rust environment installed.
kozikow 10y ago
1. Ag have nice editor integration. I would miss emacs helm-projectile-ag 2. Pcre is good regexp flavor to master. It is have good balance of speed, power and popularity. In addition to Ag, there are accessible libraries in many languages, including python. I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.
- burntsushi 10y ago
  PCRE suffers from worst case exponential behavior, so it's not suitable for all tasks. For the most part, the syntax supported by ripgrep is a strict subset of the syntax supported by PCRE. But yes, I can agree that supporting PCRE can be considered an advantage if you use advanced features heavily (backreferences and lookaround come to mind).
  - db48x 10y ago
    Meh. Back-references and lookaround both take too much brainpower to use at an interactive shell. I've used them few times in programs that I was writing, but just to find some text in some files on my disk? Never.
- pcwalton 10y ago
  > I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better. On the contrary, I think we can do better than a big pile of C code: https://www.cvedetails.com/vulnerability-list/vendor_id-3265/opdos-1/Pcre.html https://www.cvedetails.com/vulnerability-list/vendor_id-3265...
justinmayer 10y ago
Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.
- burntsushi 10y ago
  I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.
  - nkantar 10y ago
    I like ripgrep quite a bit from trying it today, and am hoping to find time to work on a Vim plugin this weekend. No promises, but I'll share as soon as I have something usable.
    - burntsushi 10y ago
      That's fantastic! Please don't hesitate to file an issue if you run into problems.
- eel 10y ago
  This is what I set in my .vimrc today, and it seems to be working pretty well: if executable('rg') set grepprg=rg\ --no-heading\ --vimgrep set grepformat=%f:%l:%c:%m endif Then you can just :grep yourquery (or :grep yourquery path), and the results will show in a quickfix window.
wamatt 10y ago
On a somewhat related note. There does not appear be a popular indexed full-text search tool in existence. Think cross-platform version of Spotlight's mdfind. Could there be something fundamental that makes this approach unsuitable for code search? Alternatively, something like locate, but realtime and fulltext, instead of filename only.
- diimdeep 10y ago
  AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.
  - wamatt 10y ago
    sure, FAT. OTOH some people think it's not a totally intractable problem A cross-platform file change monitor with multiple backends: Apple OS X File System Events, BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend. http://emcrisostomo.github.io/fswatch/ http://emcrisostomo.github.io/fswatch/ https://github.com/emcrisostomo/fswatch https://github.com/emcrisostomo/fswatch
    - diimdeep 10y ago
      nice.
_audakel 10y ago
"Anti-pitch I’d like to try to convince you why you shouldn’t use ripgrep. Often, this is far more revealing than reasons why I think you should use ripgrep." Love that he added this
h1d 10y ago
"if you like speed, saner defaults, fewer bugs and Unicode" Warning - Conditional always returns true.
wruza 10y ago
... $ rg -uu foobar # similar to `grep -r` $ rg -uuu foobar # similar to `grep -a -r` I knew it. The name is absolutely ironic. I cannot just drop-it-in and make all my scripts and whatever scripts I download work immediately faster (nor is it compatible with my shell typing reflexes). New, shiny, fast tool, doomed from birth.
- burntsushi 10y ago
  It fundamentally can't be interface compatible, sorry. I think I was pretty clear about this in the blog. :-)
cwillu 10y ago
I wish more people actually took steps to optimize disk io though; my current source tree may be in cache, but my logs certainly aren't. Nor are my /usr/share/docs/, /usr/includes/, or my old projects. Chris Mason of btrfs fame did some proof of concept work for walking and reading trees in on-disk order, showing some pretty spectacular potential gains: https://oss.oracle.com/~mason/acp/ https://oss.oracle.com/~mason/acp/ Tooling to do your own testing: https://oss.oracle.com/~mason/seekwatcher/ https://oss.oracle.com/~mason/seekwatcher/
AlisdairO 10y ago
Superb work, and a superb writeup. It's really great to see such an honest and thorough evaluation.
deleted 10y ago
[deleted]
visarga 10y ago
Great tool. Does there exist a faster implementation of sort as well? I once implemented quicksort in C and it was faster than Unix sort by a lot, I mean, seconds instead of minutes for 1 million lines of text.
- burntsushi 10y ago
  I've never waited more that a few milliseconds to sort a million lines. I expect GNU sort does quite a bit, including handling data sets that don't fit into memory. Anyway, this post is about regexes, which really has nothing at all to do with sorting. They are two very different problems. :)
chx 10y ago
I am not sure how excited I am ... I readily accept this to be faster than ag -- but ag already scans 5M lines in a second for a string literal on my machine. Not having to switch tools when I need a recursive regexp is win enough to tolerate a potential .4s vs .32s second everyday search.
TheGrassyKnoll 10y ago
Mega-Thanks to the authors of grep (longtime user) ack (nice innovation) ag (outstanding work) rg (outstanding work)
reubano 10y ago
Nice writeup! Any chance you'll support macports for those of us who never jumped ship to homebrew?
- burntsushi 10y ago
  I'm not a mac user, so I'm terribly unfamiliar with the ecosystem. In principle, I have no problem with supporting macports, but I haven't looked into it. One thing that would be a huge help is if someone briefly wrote up what would be necessary to get ripgrep into macports: https://github.com/BurntSushi/ripgrep/issues/10 https://github.com/BurntSushi/ripgrep/issues/10
glangdale 10y ago
I'm glad to see this work get written up. If anyone wants a good project, there are some optimizations we (the Hyperscan team) have done in our "Teddy" SIMD string implementation that aren't captured (yet) in the Rust implementation to my knowledge. We're very happy to see techniques from our library get used in other projects as one of the points of open sourcing it (https://github.com/01org/hyperscan https://github.com/01org/hyperscan) was to share how we do things with the community.
- burntsushi 10y ago
  You guys have so many amazing optimizations that I haven't captured yet. :-) I personally love bragging about your Teddy algorithm everywhere I go! Thank you so much for opening up the Hyperscan project. It has been a huge boon!
timonv 10y ago
Impressive! It works really well. Has anyone set it up with ctrlp in vim? I have `rg %s --files --color=never`, which works, but shows an empty white line at the prompt, so I need to use a cursor to jump to the desired file.
rob_cameron 10y ago
Nice work. But your story is incomplete if you don't include comparison with icgrep (parabix.costar.sfu.ca). Although icgrep is still an active research project, it is faster in many cases and has broader Unicode support (full Unicode level 1 of UTS #18, plus many level 2 features). For example, try the '\N{SMIL(E|ING)}' search that finds lines containing emoji characters with SMILE or SMILING in their Unicode name. icgrep also correctly applies Unicode character class intersection with expressions such as [\p{Greek}&&\p{Lu}], while ripgrep fails to meet UTS 18 level 1 requirements by interpreting '&&' as literal characters. Nevertheless, we appreciate the challenge that ripgrep presents to our performance story. We definitely see some cases in which rg achieves better performance taking advantage of fixed strings somewhere in the pattern. We'll have to work on that... But for patterns based on Unicode classes (e.g., \p{Greek}, icgrep can be much faster, especially on large files. (We are only focused on big data applications -- icgrep has significant dynamic compilation overhead). It also does very well in cases involving alternations and ambiguity. The icgrep performance story is based primarily on a new bitwise data parallel regular expression algorithm working off the Parabix transform representation of text. See our ICA3PP 2015 or PACT 2014 papers. It might be fair to say that icgrep is not yet polished enough for inclusion in your study. I just added our first implementation of -r/-R flags last night and we certainly haven't yet handled .gitignore, etc. But if you want to understand the truth about regular expression performance, I think that the data point represented by icgrep (and its continuing development) needs to be included.
- technopath 10y ago
  Or you could do it yourself: > The benchmark runner is a Python program (requires at least Python 3.5) that you can use to not only run the benchmarks themselves, but download the corpora used in the benchmarks as well. The script is called benchsuite and is in the ripgrep repository.
- burntsushi 10y ago
  I did a brief performance comparison with icgrep here: https://github.com/BurntSushi/ripgrep/issues/63 https://github.com/BurntSushi/ripgrep/issues/63 --- Yes, it does a little better on things like \p{Greek} (and has even better Unicode support than ripgrep) but is missing literal optimizations, which are a ridiculously common case for search tools like this. (As you acknowledged.) Just about every single person who has come to me with a performance comparison of ripgrep has used a simple literal pattern, because that's what people search. You need to knock that case out of the park to be competitive. Even sift and pt do this to some extent. Hyperscan is another top-notch regex implementation worth looking at too, for example. With that said, it does look like icgrep could be added to the single-file benchmark at least, but it is by far the hardest piece of software to actually get installed. (Your build instructions required me to compile llvm.) > But your story is incomplete if you don't include comparison with icgrep My story is incomplete for a very very large number of reasons. icgrep is only one of them. As I said in the blog post, the benchmarks are not only biased, but curated. Of course, I agree icgrep is worth looking into. It's on my list to learn more about. > It also does very well in cases involving alternations and ambiguity. So does ripgrep. :-) $ ls -hl OpenSubtitles2016.raw.en -rw-r--r-- 1 andrew users 9.3G Sep 10 11:51 OpenSubtitles2016.raw.en $ time rg '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m30.606s user 0m29.987s sys 0m0.567s $ time icgrep '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m41.346s user 0m40.770s sys 0m0.513s $ time rg -i '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m17.611s user 0m17.100s sys 0m0.510s $ time icgrep -i '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m43.715s user 0m43.193s sys 0m0.523s In the case insensitive search, ripgrep isn't actually doing any literal optimizations, but it is in the first case. Namely, it sees that ` ` is required in every alternate. Since ` ` is so common, this ends up slowing down the search. (This shows how literal optimizations aren't necessarily so simple, because it's easy to make a mistake like I've done and cause search to be slower.) If we change ` ` to `\s+`, we can see ripgrep regain its performance precisely because it can't do any literal optimizations: $ time rg '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m17.607s user 0m17.117s sys 0m0.490s $ time icgrep '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m46.368s user 0m45.883s sys 0m0.483s $ time rg -i '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m17.583s user 0m17.080s sys 0m0.493s $ time icgrep -i '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m48.662s user 0m48.127s sys 0m0.503s Nevertheless, ripgrep outperforms icgrep in every case. Now, with all that said, I bet this is my bias showing. I'm confident you can find other cases (we don't already know about) where icgrep beats ripgrep. :-) BTW, the corpus I used can be got here: http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.en.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl... (The one I used in my benchmark is a sample of this. For these benchmarks here, I used the full file.) The Cyrllic corpus is here: http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.ru.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl...

740 points by anp 10y ago

xuhu 10y ago

Why not make --with-filename default even for e.g. "rg somestring" ? That seems like it could hinder adoption since grep does it and it's useful when piping to other commands. Is it enabled when you specify a directory (rg somestring .) ?

burntsushi 10y ago
It should be the default whenever you search more than one file.

fsiefken 10y ago

nice, but does it compile and run on armhf? I don't see any binaries

deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
burntsushi 10y ago
I'm not terribly familiar with Rust's story on ARM, but I do know there are people working on cross compilation. I'll see if I can hook it up to the CI later today. Certainly, there's nothing in ripgrep itself that should prevent it from working on ARM.
- echelon 10y ago
  I cross compile my home automation sever that I wrote in Rust to ARM. It should work.
  - Cshelton 10y ago
    I'll be doing that soon! Care to share your code? or explain what exactly you're doing with it?
    - echelon 10y ago
      I've published a few crates in the home automation space, but they're both pretty terrible at this point. One of them is for controlling WeMo devices, and now that I know Rust better I'm going to rewrite it using the new blocking APIs rather than Mio. (And real SOAP support.) What kind of home automation will you be doing? I'd absolutely love to collaborate on a gateway that interacts with Wemo, Z-wave, Zigbee, etc. and can do basic things like events, scenes, schedules, etc. I haven't focused on home automation all summer long since I've been trying to get my Donald Trump TTS engine (also written in Rust) ready for the election. It's soooo close.
mbrubeck 10y ago
I was able to build ripgrep from source for ARM, cross-compiling from my laptop running Debian, following the directions here: https://github.com/japaric/rust-cross#tldr-ubuntu-example https://github.com/japaric/rust-cross#tldr-ubuntu-example (I haven't actually run it because I don't have an ARM linux device handy.)
- pyroholic 10y ago
  Cross-compiled and ran a couple basic searches on an armv7l device. So at least the basic functionality works just fine.

dikaiosune 10y ago

Compiling it to try right now... Some discussion over on /r/rust: https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_faster_than_grep_ag_git_grep_ucg_pt/ https://www.reddit.com/r/rust/comments/544hnk/ripgrep_is_fas... EDIT: The machine I'm on is much less beefy than the benchmark machines, which means that the speed difference is quite noticeable for me.

echelon 10y ago

Rust is really staring to be seen in the wild now.

CJefferson 10y ago
I agree, and I'm already using both ripgrep and rust-parallel ( https://github.com/mmstick/parallel https://github.com/mmstick/parallel , a gnu-parallel replacement which should probably get another name ). I am really happy to see Rust apps actually been written -- Rust programmers seem to be actually trying to replace the existing code lying around, instead of just insulting it and telling us how much better the world would be if we used their language.
- ekidd 10y ago
  Honestly, for a certain kind of developer—I haven't fingerpointed exactly which kind—writing Rust code is dangerously addictive. There's some combination of aggressive type checking, tooling, and expressiveness that just hits a sweet spot in my brain. Rust fills roughly the same niche in my brain as old-school C++, except it also lets me do some rudimentary functional stuff, there are virtually no footguns, and the CLI tooling is out of this world. And I've long since made friends with the borrow checker. So it's tempting to just spend my life cranking out Rust code as fast as I can. It's actually a problem. :-)
  - CJefferson 10y ago
    Well in the interests of making contraversal statements, that's now two Rust programs I plan on using day-to-day, and zero for Haskell, Scala, and I'm sure a few other languages I'm not remembering.
    - unfamiliar 10y ago
      It will only be a matter of time before someone rewrites xmonad in Rust.
      - 10y ago

pixelbeat 10y ago

Thanks for the detailed comparisons and writeup. I find this simple wrapper around grep(1) very fast and useful: http://www.pixelbeat.org/scripts/findrepo http://www.pixelbeat.org/scripts/findrepo

burntsushi 10y ago
Thanks for the kind words! Note that the key thing that `findrepo` doesn't support is respecting your .gitignore files. For example, in the Rust ecosystem, we often have a `target` directory in our projects that contains a lot of stuff we probably don't want to search. In fact, running `cargo new` will add that directory to your `.gitignore` automatically! Tools like The Silver Searcher and ripgrep will ignore that directory (and all others like it) automatically. There are other advantages to ripgrep. For example, every other tool that supports Unicode as well as ripgrep (that's `git grep` and GNU grep) experience a substantial slow down when trying to use more advanced Unicode features (like \w, -i, etc.). This is one of the things my benchmarks show.
- pixelbeat 10y ago
  Yes ignoring .gitignore contents is a very useful feature. That could be added quite easily to findrepo though would probably have to be an opt as it is often useful to search intermediate build files etc. that aren't checked in. Whereas `git grep` handles the other use case of only searching checked in files. Interesting info wrt efficient unicode processing for \w and -i. cheers

cm3 10y ago

To build a static Linux binary with SIMD support, run this: RUSTFLAGS="-C target-cpu=native" rustup run nightly cargo build --target x86_64-unknown-linux-musl --release --features simd-accel

burntsushi 10y ago
That's an awesome demonstration of how easy it is to swap out the libc in Rust. :-) Note that I also distribute statically compiled executables with musl and SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
- cm3 10y ago
  > That's an awesome demonstration of how easy it is to swap out the libc in Rust. :-) Now I have to look up how I use cargo to build a static binary on FreeBSD, where I don't have to swap out libc. > Note that I also distribute statically compiled executables with musl and SIMD enabled (using target-feature=+ssse3 instead of target-cpu=native): https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases I took the flag from your blog post, thanks for pointing out the explicit feature flag. That will allow the binary to run on more cpus.

jonstewart 10y ago

Nice! Lightgrep[1] uses libicu et al to look up code points for a user-specified encoding and encode them as bytes, then just searches for the bytes. Since ripgrep is presumably looking just for bytes, too, and compiling UTF-8 multibyte code points to a sequence of bytes, perhaps you can do likewise with ICU and support other encodings. ICU is a bear to build against when cross-compiling, but it knows hundreds of encodings, all of the proper code point names, character classes, named properties, etc., and the surface area of its API that's required for such usage is still pretty small. [1]: http://strozfriedberg.github.io/liblightgrep http://strozfriedberg.github.io/liblightgrep

burntsushi 10y ago
I hadn't heard of liblightgrep, nice. It's on my short list for looking more closely. I doubt I'd ever be comfortable with Rust's regex engine growing a dependency on libicu, but it's still worth understanding your implementation. Some questions, if you don't mind. The big one is: does your regex engine use finite automata, and does it put the text decoding into the automaton itself? For example, when you compile the `.` regex, you end up with an automaton that inlines UTF-8 decoding itself. It looks like this: https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fabbd0a3 https://gist.github.com/anonymous/8fbe170bfcca5d7475b59299fa... Does you regex library do that for each type of encoding? Or is there a transcoding step?
- jonstewart 10y ago
  Reply fail, see above.
jonstewart 10y ago
Yep! It's a multipattern engine so we use an NFA--slower than a normal engine, but effective when you have thousands of patterns to search for in binary streams. Since we are often searching binary, we'll encounter text fragments in lots of different encodings, so they're all compiled into the NFA. We wrote a paper about its Unicode support: http://www.dfrws.org/sites/default/files/session-files/paper-unicode_search_of_dirty_data_or_how_i_learned_to_stop_worrying_and_love_unicode_technical_standard_18.pdf http://www.dfrws.org/sites/default/files/session-files/paper....

qwertyuiop924 10y ago

That is really cool. Although I think this is a case where Good Enough will beat amazing, at least for me (especially given how much I use backrefs).

spicyj 10y ago

rg is harder to type with one hand because it uses the same finger twice. :)

pixelbath 10y ago
But 'grep' doesn't?
- echelon 10y ago
  True, but `ag` and `ack` don't. This is a really interesting point. If we're going to be typing commands frequently, it would make sense for them to be optimally efficient to type using QWERTY (and maybe other common keyboard layouts). `ls` and `cd` are super easy to type.
adelarsq 10y ago
lol
robinson-wall 10y ago
Off topic, but you've just made me realise I don't always use the same finger when I type the letter R. If I were typing lower case "rg" I use middle finger for r and then index finger for g. I'm now thinking about it too much to be able to type naturally, but I seem to use middle and index interchangeably for some keys depending on which key was just pressed / is next to be pressed. ... and I just noticed if I sometimes use different fingers from the "wrong" hand to type depending on context too! I've typed the letter Y using right-index, left-index and left-middle and left-index during the course of writing this post. I guess this is what learning to touch type without being actually taught buys you :)
- auganov 10y ago
  Was just about to write a similar reply too. I type "rg" the same way. My left hand very often goes for the YHN column, the right hand won't ever touch TGB though. For typing "lol" I'll very often just move my right hand to the right (or sort of angle it) and press the l with the index and the o with the middle finger. I've been plateaued at a 110wpm avg (121 top speed) @ 10fastfingers for quite a few months now, so perhaps breaking some of these bad habits would help me get to the desired 140.

chalana 10y ago

I'm never sure whether or not I should adopt these fancy new command line tools that come out. I get them on my muscle memory and then all of a sudden I ssh into a machine that doesn't have any of these and I'm screwed...

petdance 10y ago
I have to switch back and forth between ack and grep all the time. Sometimes I use ack, sometimes I use grep. I wrote ack, but I've never stopped using grep, and ack has never been intended as a replacement for grep. For most of the common ack flags, they are the same as grep. -i, -l, -v, -w, -A, -B, -C, etc. That was intentional, to minimize that whiplash. One other suggestion is that you drop ack into your ~/bin directory that you sync between machines. It's a single Perl file, and that portability is a feature. As long as your machine has a Perl on it, you can use ack.
sitkack 10y ago
Just make your ssh script install all the binaries where ever you go! ssh-caravan

losvedir 10y ago

Meh, yet another grep tool.... wait, by burntsushi! Whenever I hear of someone wanting to improve grep I think of the classic ridiculous fish piece[0]. But when I saw that this one was by the author of rust's regex tools, which I know from a previous post on here, are quite sophisticated, I perked up. Also, the tool aside, this blog post should be held up as the gold standard of what gets posted to hacker news: detailed, technical, interesting. Thanks for your hard work! Looking forward to taking this for a spin. [0] http://ridiculousfish.com/blog/posts/old-age-and-treachery.html http://ridiculousfish.com/blog/posts/old-age-and-treachery.h...

trishume 10y ago
Totally agree. This is like burntsushi's technical tour de force, the culmination of years of work building up a stack of technology leading up to this utility and article. It's a simple tool, but he's built every piece from the ground up to be the very best, and then documented the shit out of it. So cool.
pohl 10y ago
Another burntsushi project was recently posted, but didn't get much attention: https://news.ycombinator.com/item?id=12559515 https://news.ycombinator.com/item?id=12559515
- skanga 10y ago
  Looks interesting. But I did not find binaries for it and do not want to setup a rust env to try it out.
  - luckman212 10y ago
    The releases are right there on GitHub: https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
    - burntsushi 10y ago
      I think the GP was asking about xsv, not ripgrep. There are binary releases for xsv though: https://github.com/BurntSushi/xsv/releases https://github.com/BurntSushi/xsv/releases

bodyfour 10y ago

It would be interesting to benchmark how much mmap hurts when operating in a non-parallel mode. I think a lot of the residual love for mmap is because it actually did give decent results back when single core machines were the norm. However, once your program becomes multithreaded it imposes a lot of hidden synchronization costs, especially on munmap(). The fastest option might well be to use mmap sometimes but have a collection of single-thread processes instead of a single multi-threaded one so that their VM maps aren't shared. However, this significantly complicates the work-sharing and output-merging stages. If you want to keep all the benefits you'd need a shared-memory area and do manual allocation inside it for all common data which would be a lot of work. It might also be that mmap is a loss these days even for single-threaded... I don't know. Side note: when I last looked at this problem (on Solaris, 20ish years ago) one trick I used when mmap'ing was to skip the "madvise(MADV_SEQUENTIAL)" if the file size was below some threshold. If the file was small enough to be completely be prefetched from disk it had no effect and was just a wasted syscall. On larger files it seemed to help, though.

karmakaze 10y ago
I recently questioned if/why parallel mmap might be slower without a satisfactory conclusion. One specific thing I couldn't answer is if reusing the same filesystem buffer and program memory addresses has a less negative effect than reading a wide range of mapped memory addresses.
- brandmeyer 10y ago
  Since all processors must share the mapping, - The initial mapping of each file in any thread must halt all of the threads which are otherwise active. - Every page fault in any mapping must also halt all of the threads. Worse, since the page tables are getting munged, some or all of the TLB cache is getting flushed every time, again, on every processor. I'm not sure of the details, but this hypothesis should be directly testable. IIRC, there are some hardware performance counters for time spent waiting on TLB lookups. Addendum: One other possibility is that the mere act of extending the working set size (of the address space) is blowing the TLB cache.

petre 10y ago

Does it use PCRE (not the lib, the regex style). If not, ack is just fine. My main concern with grep are Posix regular expressions.

steveklabnik 10y ago
"regex style" is a bit too broad. The regexes feel more like PCRE syntactically than they do POSIX, but does not support the more exotic features of PCRE.
Occivink 10y ago
It uses the re2 style, as provided by the rust regex library.

Tim61 10y ago

I love the layout of this article. Especially the pitch and anti-pitch. I wish more more tools/libraries/things would make note of their downsides. I'm convinced to give it a try.

minimax 10y ago

In contrast, GNU grep uses libc’s memchr, which is standard C code with no explicit use of SIMD instructions. However, that C code will be autovectorized to use xmm registers and SIMD instructions, which are half the size of ymm registers. I don't think this is correct. glibc has architecture specific hand rolled (or unrolled if you will lol) assembly for x64 memchr. See here: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/memrchr.S;h=840de30cd71ba96b3ae43540e6ac255c28906cc5;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86...

burntsushi 10y ago
Drats, you're totally right. It's easy to mess up that kind of thing. Thankfully, it looks like my analysis remains mostly unchanged. I don't see any AVX2 in there (and indeed, I didn't when I looked at the profile either, in contrast to Go's implementation). I updated the blog, thanks again for the clarification.

libman 10y ago

Tragically the news that LLVM is switching to a non-Copyfree license (see copyfree.org/standard/rejected) has ruined everything... Nothing written in Rust can be called Free Software. :(

alphapapa 10y ago
According to that site, Free Software (as defined by the FSF) is not Copyfree. As far as I can tell, Copyfree is like the BSD license, except you can only restrict changes to the license text itself; the license itself may not be changed or added to, but everything else in the project may be used and changed in any way. So, effectively, it seems like a license where the license itself is copyrighted with all rights reserved, but the rest of the project is public domain. So, just like BSD, anyone can take a Copyfree project proprietary, turning it non-free. The GPL is as important today as ever.

krylon 10y ago

When I use grep (which is fairly regularly), the bottleneck is nearly always the disk or the network (in case of NFS/SMB volumes). Just out of curiosity, what kind of use case makes grep and prospective replacements scream? The most "hardcore" I got with grep was digging through a few gigabytes of ShamePoint logs looking for those correlation IDs, and that apparently was completely I/O-bound, the CPUs on that machine stayed nearly idle.

nolemurs 10y ago
`grep` itself is admirably fast on single files, and if you're just searching a small number of files, the bottleneck will be IO, and `grep` will be about as fast as anything. But if you want to use `grep` recursively on a selected subset of millions of files it suffers a bit. `grep` assumes that if you want to collect some subset of files to search, you'll do that with some other utility, and pass the list of files to `grep`, but that creates a bottleneck. More recent search programs like `ack` and `ag` solve the multifile bottleneck issue, but also perform worse than `grep` on single large files.
- krylon 10y ago
  Thank you!
- burntsushi 10y ago
  > and `grep` will be about as fast as anything I'd encourage you to check out the benchmarks in my blog, especially the subtitle benchmarks, because this isn't actually true in the case of ripgrep. :-)
  - nolemurs 10y ago
    I take it back! I'm definitely switching over to using ripgrep for the next bit. Off topic - is there a way to exclude directories? I tried doing things like `rg pattern -g '!my_dir/' but nothing I tried seemed to work.
    - 10y ago

pmontra 10y ago

It looks very good and I'd like to try it. However I'm lazy and I don't want to install all the Rust dev environment to compile it. Did anybody build a .deb for Ubuntu 16?

steveklabnik 10y ago
If you don't want to compile it yourself, the blog post has links to binaries. Since they're entirely self-contained, you don't need a full .deb to compile them; just delete the binary when you want to get rid of it.
- pmontra 10y ago
  I didn't notice that. Thanks. It works. The binary is 11 MB, or 1.5 MB stripped. ag is 69176 bytes. Shared libraries were a good invention :-) Sooner or later rg will be packaged properly too.
  - steveklabnik 10y ago
    Yes, especially given that rustc/cargo are appearing in distro's package sets already. Exciting times :)
  - cgag 10y ago
    I'd much rather have a self contained binary than 11 more MB of free space.

serge2k 10y ago

> We will attempt to do the impossible Oh well. Waste of time then.

bo1024 10y ago
Maybe the author hasn't had breakfast yet and has only done 5 impossible things so far?

zatkin 10y ago

>It is not, strictly speaking, an interface compatible “drop-in” replacement for both, but the feature sets are far more similar than different.

lobster_johnson 10y ago

Very nice. Not only fast, but feels modern. Tried it out on a 3.5GB JSON file: # rg rg erzg4 k.json > /dev/null 1.80s user 2.54s system 53% cpu 8.053 total # rg with 4 threads rg -j4 erzg4 k.json > /dev/null 1.76s user 1.29s system 99% cpu 3.059 total # OS X grep grep erzg4 k.json > /dev/null 60.62s user 0.96s system 99% cpu 1:01.75 total # GNU Grep ggrep erzg4 k.json > /dev/null 1.96s user 1.43s system 88% cpu 2.691 total GNU Grep wins, but it's pretty crusty, especially with regards to its output (even with colourization).

burntsushi 10y ago
My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory. This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and didn't record any measurements. I then ran them another 10 times in which I recorded them. You can see the raw output here: https://github.com/BurntSushi/ripgrep/blob/master/benchsuite/runs/2016-09-20-ubuntu1604-ec2/raw.csv https://github.com/BurntSushi/ripgrep/blob/master/benchsuite... In any case, on my underpowered Mac, here are some results on a 1.2 GB file (notice how much the time fluctuates until its fully in cache): mac:~ andrew$ ggrep --version ggrep (GNU grep) 2.25 Packaged by Homebrew mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m5.447s user 0m0.600s sys 0m0.350s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m1.247s user 0m0.549s sys 0m0.264s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.803s user 0m0.542s sys 0m0.259s mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.805s user 0m0.544s sys 0m0.260s And now for rg: mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m1.062s user 0m0.339s sys 0m0.333s mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.640s user 0m0.337s sys 0m0.302s mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null real 0m0.637s user 0m0.336s sys 0m0.300s Oh! And check this out, on a Mac, not using a memory map for single files is faster. My goodness---memory map performance is all over the place. mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl --no-mmap > /dev/null real 0m0.445s user 0m0.170s sys 0m0.274s If I do this on my Linux machine on the same file, I get timings of 0.275s for rg, 0.398s for rg with no memory maps (opposite direction for Mac) and 0.708s for GNU grep (v 2.25). Benchmarks are fun, eh?

ggreer 10y ago

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support. Switching would cause me to receive a lot of angry emails. Maybe I'll do it anyway. PCRE has some annoying limitations. (For example, it can only search up to 2GB at a time.) 2. Not counting line numbers by default. The blog post addresses this, but I think results without line numbers are far less useful; so much so that I've traded away performance in ag. (Note that even if you tell ag not to print line numbers, it still wastes time counting them. The printing code is the result of me merging a lot of PRs that I really shouldn't have.) 3. Not using mmap(). This is a big one, and I'm not sure what the deal is here. I just added a --nommap option to ag in master.[1] It's a naive implementation, but it benchmarks comparably to the default mmap() behavior. I'm really hoping there's a flag I can pass to mmap() or madvise() that says, "Don't worry about all that synchronization stuff. I just want to read these bytes sequentially. I'm OK with undefined behavior if something else changes the file while I'm reading it." The author also points out correctness issues with ag. Ag doesn't fully support .gitiginore. It doesn't support unicode. Inverse matching (-v) can be crazy slow. These shortcomings are mostly because I originally wrote ag for myself. If I didn't use certain gitignore rules or non-ASCII encodings, I didn't write the code to support them. Some expectation management: If you try out ripgrep, don't get your hopes up. Unless you're searching some really big codebases, you won't notice the speed difference. What you will notice, however, are the feature differences. Take a look at https://github.com/BurntSushi/ripgrep/issues https://github.com/BurntSushi/ripgrep/issues to get a taste of what's missing or broken. It will be some time before all those little details are ironed-out. That said, may the best code searching tool win. :) 1. https://github.com/ggreer/the_silver_searcher/commit/bd65e2691d6ce673e026842d7e7d02bc25adfd26 https://github.com/ggreer/the_silver_searcher/commit/bd65e26...

burntsushi 10y ago
Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory maps on Windows where they seem to do a bit better. (I think my blog post gives enough details that you could reproduce the benchmark environment precisely if you were so inclined. This was important to me, so I spent a lot of time documenting it.) 3. The set of features supported by rg should be very very close to what is supported by ag. Reviewing `ag`'s man page again, probably the only things missing from rg are --ackmate, --depth, some of the color configurability flags (but rg does do coloring), --passthrough, --smart-case and --stats maybe? I might be missing some others. And Mercurial support (but ag's is incomplete). In exchange, rg gives you much better single file performance, better large-repo performance and real Unicode support that doesn't slow way down. I'd say those are pretty decent expectations. :-) Thanks for ag by the way. It and ack have definitely triggered a new kind of searching. I have some further information retrievalish ideas on evolving the concept, but those will have to wait!

hxn 10y ago

Looks like every tool has its upsides and downsides. This one lacks full PCRE syntax support. Does one have to install Rust to use it?

kod 10y ago
No. binaries: http://blog.burntsushi.net/ripgrep/#installation http://blog.burntsushi.net/ripgrep/#installation
heinrich5991 10y ago
No, Rust tools don't need a Rust environment installed.

kozikow 10y ago

1. Ag have nice editor integration. I would miss emacs helm-projectile-ag 2. Pcre is good regexp flavor to master. It is have good balance of speed, power and popularity. In addition to Ag, there are accessible libraries in many languages, including python. I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.

burntsushi 10y ago
PCRE suffers from worst case exponential behavior, so it's not suitable for all tasks. For the most part, the syntax supported by ripgrep is a strict subset of the syntax supported by PCRE. But yes, I can agree that supporting PCRE can be considered an advantage if you use advanced features heavily (backreferences and lookaround come to mind).
- db48x 10y ago
  Meh. Back-references and lookaround both take too much brainpower to use at an interactive shell. I've used them few times in programs that I was writing, but just to find some text in some files on my disk? Never.
pcwalton 10y ago
> I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better. On the contrary, I think we can do better than a big pile of C code: https://www.cvedetails.com/vulnerability-list/vendor_id-3265/opdos-1/Pcre.html https://www.cvedetails.com/vulnerability-list/vendor_id-3265...

justinmayer 10y ago

Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.

burntsushi 10y ago
I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.
- nkantar 10y ago
  I like ripgrep quite a bit from trying it today, and am hoping to find time to work on a Vim plugin this weekend. No promises, but I'll share as soon as I have something usable.
  - burntsushi 10y ago
    That's fantastic! Please don't hesitate to file an issue if you run into problems.
eel 10y ago
This is what I set in my .vimrc today, and it seems to be working pretty well: if executable('rg') set grepprg=rg\ --no-heading\ --vimgrep set grepformat=%f:%l:%c:%m endif Then you can just :grep yourquery (or :grep yourquery path), and the results will show in a quickfix window.

wamatt 10y ago

On a somewhat related note. There does not appear be a popular indexed full-text search tool in existence. Think cross-platform version of Spotlight's mdfind. Could there be something fundamental that makes this approach unsuitable for code search? Alternatively, something like locate, but realtime and fulltext, instead of filename only.

diimdeep 10y ago
AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.
- wamatt 10y ago
  sure, FAT. OTOH some people think it's not a totally intractable problem A cross-platform file change monitor with multiple backends: Apple OS X File System Events, BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend. http://emcrisostomo.github.io/fswatch/ http://emcrisostomo.github.io/fswatch/ https://github.com/emcrisostomo/fswatch https://github.com/emcrisostomo/fswatch
  - diimdeep 10y ago
    nice.

_audakel 10y ago

"Anti-pitch I’d like to try to convince you why you shouldn’t use ripgrep. Often, this is far more revealing than reasons why I think you should use ripgrep." Love that he added this

h1d 10y ago

"if you like speed, saner defaults, fewer bugs and Unicode" Warning - Conditional always returns true.

wruza 10y ago

... $ rg -uu foobar # similar to `grep -r` $ rg -uuu foobar # similar to `grep -a -r` I knew it. The name is absolutely ironic. I cannot just drop-it-in and make all my scripts and whatever scripts I download work immediately faster (nor is it compatible with my shell typing reflexes). New, shiny, fast tool, doomed from birth.

burntsushi 10y ago
It fundamentally can't be interface compatible, sorry. I think I was pretty clear about this in the blog. :-)

cwillu 10y ago

I wish more people actually took steps to optimize disk io though; my current source tree may be in cache, but my logs certainly aren't. Nor are my /usr/share/docs/, /usr/includes/, or my old projects. Chris Mason of btrfs fame did some proof of concept work for walking and reading trees in on-disk order, showing some pretty spectacular potential gains: https://oss.oracle.com/~mason/acp/ https://oss.oracle.com/~mason/acp/ Tooling to do your own testing: https://oss.oracle.com/~mason/seekwatcher/ https://oss.oracle.com/~mason/seekwatcher/

AlisdairO 10y ago

Superb work, and a superb writeup. It's really great to see such an honest and thorough evaluation.

deleted 10y ago

[deleted]

visarga 10y ago

Great tool. Does there exist a faster implementation of sort as well? I once implemented quicksort in C and it was faster than Unix sort by a lot, I mean, seconds instead of minutes for 1 million lines of text.

burntsushi 10y ago
I've never waited more that a few milliseconds to sort a million lines. I expect GNU sort does quite a bit, including handling data sets that don't fit into memory. Anyway, this post is about regexes, which really has nothing at all to do with sorting. They are two very different problems. :)

chx 10y ago

I am not sure how excited I am ... I readily accept this to be faster than ag -- but ag already scans 5M lines in a second for a string literal on my machine. Not having to switch tools when I need a recursive regexp is win enough to tolerate a potential .4s vs .32s second everyday search.

TheGrassyKnoll 10y ago

Mega-Thanks to the authors of grep (longtime user) ack (nice innovation) ag (outstanding work) rg (outstanding work)

reubano 10y ago

Nice writeup! Any chance you'll support macports for those of us who never jumped ship to homebrew?

burntsushi 10y ago
I'm not a mac user, so I'm terribly unfamiliar with the ecosystem. In principle, I have no problem with supporting macports, but I haven't looked into it. One thing that would be a huge help is if someone briefly wrote up what would be necessary to get ripgrep into macports: https://github.com/BurntSushi/ripgrep/issues/10 https://github.com/BurntSushi/ripgrep/issues/10

glangdale 10y ago

I'm glad to see this work get written up. If anyone wants a good project, there are some optimizations we (the Hyperscan team) have done in our "Teddy" SIMD string implementation that aren't captured (yet) in the Rust implementation to my knowledge. We're very happy to see techniques from our library get used in other projects as one of the points of open sourcing it (https://github.com/01org/hyperscan https://github.com/01org/hyperscan) was to share how we do things with the community.

burntsushi 10y ago
You guys have so many amazing optimizations that I haven't captured yet. :-) I personally love bragging about your Teddy algorithm everywhere I go! Thank you so much for opening up the Hyperscan project. It has been a huge boon!

timonv 10y ago

Impressive! It works really well. Has anyone set it up with ctrlp in vim? I have `rg %s --files --color=never`, which works, but shows an empty white line at the prompt, so I need to use a cursor to jump to the desired file.

rob_cameron 10y ago

Nice work. But your story is incomplete if you don't include comparison with icgrep (parabix.costar.sfu.ca). Although icgrep is still an active research project, it is faster in many cases and has broader Unicode support (full Unicode level 1 of UTS #18, plus many level 2 features). For example, try the '\N{SMIL(E|ING)}' search that finds lines containing emoji characters with SMILE or SMILING in their Unicode name. icgrep also correctly applies Unicode character class intersection with expressions such as [\p{Greek}&&\p{Lu}], while ripgrep fails to meet UTS 18 level 1 requirements by interpreting '&&' as literal characters. Nevertheless, we appreciate the challenge that ripgrep presents to our performance story. We definitely see some cases in which rg achieves better performance taking advantage of fixed strings somewhere in the pattern. We'll have to work on that... But for patterns based on Unicode classes (e.g., \p{Greek}, icgrep can be much faster, especially on large files. (We are only focused on big data applications -- icgrep has significant dynamic compilation overhead). It also does very well in cases involving alternations and ambiguity. The icgrep performance story is based primarily on a new bitwise data parallel regular expression algorithm working off the Parabix transform representation of text. See our ICA3PP 2015 or PACT 2014 papers. It might be fair to say that icgrep is not yet polished enough for inclusion in your study. I just added our first implementation of -r/-R flags last night and we certainly haven't yet handled .gitignore, etc. But if you want to understand the truth about regular expression performance, I think that the data point represented by icgrep (and its continuing development) needs to be included.

technopath 10y ago
Or you could do it yourself: > The benchmark runner is a Python program (requires at least Python 3.5) that you can use to not only run the benchmarks themselves, but download the corpora used in the benchmarks as well. The script is called benchsuite and is in the ripgrep repository.
burntsushi 10y ago
I did a brief performance comparison with icgrep here: https://github.com/BurntSushi/ripgrep/issues/63 https://github.com/BurntSushi/ripgrep/issues/63 --- Yes, it does a little better on things like \p{Greek} (and has even better Unicode support than ripgrep) but is missing literal optimizations, which are a ridiculously common case for search tools like this. (As you acknowledged.) Just about every single person who has come to me with a performance comparison of ripgrep has used a simple literal pattern, because that's what people search. You need to knock that case out of the park to be competitive. Even sift and pt do this to some extent. Hyperscan is another top-notch regex implementation worth looking at too, for example. With that said, it does look like icgrep could be added to the single-file benchmark at least, but it is by far the hardest piece of software to actually get installed. (Your build instructions required me to compile llvm.) > But your story is incomplete if you don't include comparison with icgrep My story is incomplete for a very very large number of reasons. icgrep is only one of them. As I said in the blog post, the benchmarks are not only biased, but curated. Of course, I agree icgrep is worth looking into. It's on my list to learn more about. > It also does very well in cases involving alternations and ambiguity. So does ripgrep. :-) $ ls -hl OpenSubtitles2016.raw.en -rw-r--r-- 1 andrew users 9.3G Sep 10 11:51 OpenSubtitles2016.raw.en $ time rg '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m30.606s user 0m29.987s sys 0m0.567s $ time icgrep '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m41.346s user 0m40.770s sys 0m0.513s $ time rg -i '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m17.611s user 0m17.100s sys 0m0.510s $ time icgrep -i '\w+ Holmes|\w+ Watson|\w+ Adler|\w+ Moriarty|\w+ Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m43.715s user 0m43.193s sys 0m0.523s In the case insensitive search, ripgrep isn't actually doing any literal optimizations, but it is in the first case. Namely, it sees that ` ` is required in every alternate. Since ` ` is so common, this ends up slowing down the search. (This shows how literal optimizations aren't necessarily so simple, because it's easy to make a mistake like I've done and cause search to be slower.) If we change ` ` to `\s+`, we can see ripgrep regain its performance precisely because it can't do any literal optimizations: $ time rg '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m17.607s user 0m17.117s sys 0m0.490s $ time icgrep '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 26464 real 0m46.368s user 0m45.883s sys 0m0.483s $ time rg -i '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m17.583s user 0m17.080s sys 0m0.493s $ time icgrep -i '\w+\s+Holmes|\w+\s+Watson|\w+\s+Adler|\w+\s+Moriarty|\w+\s+Lestrade' OpenSubtitles2016.raw.en | wc -l 27370 real 0m48.662s user 0m48.127s sys 0m0.503s Nevertheless, ripgrep outperforms icgrep in every case. Now, with all that said, I bet this is my bias showing. I'm confident you can find other cases (we don't already know about) where icgrep beats ripgrep. :-) BTW, the corpus I used can be got here: http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.en.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl... (The one I used in my benchmark is a sample of this. For these benchmarks here, I used the full file.) The Cyrllic corpus is here: http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.ru.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl...