10 ms·

Ripgrep 11 Released

410 points by burntsushi 8y ago

xrouge 8y ago
Is there a way to output the results in Visual Studio format, which is <filename>(<line number>)<result>? This way when I hook up ripgrep as an External Tool, the line shown in the Output window can be clicked on to open the file and go to the specific line. I see in the source code that the separator is ":"; VS expects () around the line number.
- burntsushi 8y ago
  Not natively, no. You could probably write a simple script that transforms the output into what you want though. For example, on Unix at least, rg -n pattern | sed 's/:/(/' | sed 's/:/)/' That won't work in every case (e.g., in the case of a file path containining a `:`), but it might get you pretty far. And it could probably be made more precise. Otherwise, you could always use ripgrep's --json output mode to craft its output into whatever format you like, but that's probably more work. :-) The JSON format is documented here: https://docs.rs/grep-printer/0.1.2/grep_printer/struct.JSON.html https://docs.rs/grep-printer/0.1.2/grep_printer/struct.JSON....
j0e1 8y ago
I love ripgrep. For me it has done an exceptional job at searching and finding a large list of search strings over pretty large text files. Tangentially, it is because of great tools like these written in Rust that I have major respect for the language.
dfabulich 8y ago
If you've been holding back on Ripgrep because of its 0.x version numbering, now's the time to adopt it. Ripgrep is faster than grep, faster than ag (the Silver Searcher), and even faster than git grep. The author breaks down all of these details in this 17,000 word blog post, full of detailed benchmarks. https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ (It's from 2016, but it's still mostly in good shape; ripgrep has only gotten faster since then!)
- loeg 8y ago
  The blog post is good, and fair at the time. One caution I would add is that time has not stopped for the other projects, either. I believe ag (the silver searcher) has also seen considerable performance improvements since 2016.
  - burntsushi 8y ago
    > I believe ag (the silver searcher) has also seen considerable performance improvements since 2016. Do you know what they are? I haven't noticed them, although it has been a while since I've rigorously benchmarked it. From looking at the source code, I don't think much has changed. e.g., it appears to still be using memory maps on Linux when searching large directories.
    - BeeOnRope 8y ago
      Out of curiosity, is there a better approach than memory maps?
      - shaklee3 8y ago
        The author goes over that here: https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
      - hungnmsu 8y ago
        Using mmap is generally slower than reading chunk of data and mmap does not work well if you have to handle huge files with several GBs of text data. You can find performance benchmark results and a detail analysis from @burntsushi blog and Lemire's blog post (https://lemire.me/blog/2012/06/26/which-is-fastest-read-fread-ifstream-or-mmap/ https://lemire.me/blog/2012/06/26/which-is-fastest-read-frea...).
  - hungnmsu 8y ago
    ag is slower than ripgrep in all of my benchmarks (https://github.com/hungptit/fastgrep https://github.com/hungptit/fastgrep). And ag might be slower than GNU grep if files are stored in fast storage devices i.e SSD drives even though GNU grep is a single thread command.
sqs 8y ago
Ripgrep is awesome. Thank you for making it! In addition to being really fast, it "just works". By that I mean it automatically excludes the files I want to be excluded, like `.gitignore`d files and binary files. I know I can configure ack-grep (ag) and other tools to do that, but not needing to configure it is nice. BTW, if anyone hasn't read https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/, highly recommended. It's about how ripgrep is so fast. (Edit: Just saw another comment mentioned this, too. Goes to show that making a single high-quality blog post has a big impact.)
- glangdale 8y ago
  It is a very good blog post. Recommended! Now, I will do my obligatory "burntsushi needs to clag the rest of the Teddy code" post which I must do on every discussion of ripgrep. :-) The "subtitles_alternate_casei" examples would be a good benchmark - these 48 strings should not overwhelm Teddy as they could be sensibly merged into Teddy's 8 "buckets" (in fact, they could be merged into 5 buckets) using the simple greedy merging strategy in the original Teddy implementation. This would probably be a quite good project for someone who wants to contribute to ripgrep and could likely get some nice performance wins...
  - burntsushi 8y ago
    I agree, this could be a nice project! If anyone wants to work on it, this is the place to start: https://github.com/rust-lang/regex/blob/master/src/literal/teddy_ssse3/imp.rs https://github.com/rust-lang/regex/blob/master/src/literal/t... I'll get to it myself eventually if someone else doesn't, but it will likely be a while.
    - glangdale 8y ago
      I've been thinking about a Teddy successor (which I suppose needs to be called "Taft"). Teddy is very much 'of its time' (SSSE3) and there are a lot of new approaches that seem interesting (AVX512 of Skylake generation, VBMI, Sunny Cove's even bigger slate of instructions, ARM NEON, SVE). I also have better ideas about followup confirm than I used to. There are also some prospect to pick a 'fragment' out of the whole string within Teddy or equivalent at a position not strictly at its suffix - this can even be done with ordering preserved if you are careful not to make fragment choices that allow o-o-o matches (only possible if strings overlap). I might do a bit of work on this, but I'm a bit jaded on string matching and regex matching after 13 years.
nojvek 8y ago
Vscode integrates ripgrep by default in their “search project” feature. I mostly use Vscode because of its fast search, decent auto complete and simple editing experience. Whoever made ripgrep, deserves massive kudos.
- fiddlerwoaroof 8y ago
  The OP is by the author of ripgrep
- carlmr 8y ago
  Same reason for switching for me. The projects at my work needed 2 minutes for a single regex search, with ripgrep it went to 3 seconds.
- aashcan 8y ago
  Atom's fuzzy finder also uses ripgrep as of 1.37.0 [0] https://github.com/atom/fuzzy-finder/pull/369 https://github.com/atom/fuzzy-finder/pull/369
keithnz 8y ago
version 0.10.0 to 11.0.0 is a big version leap!
- ksherlock 8y ago
  emacs did that too - 1.12 to 13.
  - cpeterso 8y ago
    Solaris jumped from 2.6 to 7 (the "2." is now implicit). Of course, Windows NT jumped from 4.0 to 2000. ;)
    - LeonidasXIV 8y ago
      But it didn't, really. NT 4.0 was followed by NT 5.0 (branded Windows 2000) and then by 5.1 (Windows XP) and then by 6.0 (Windows 7).
      - signal11 8y ago
        Vista was 6.0. Windows 7 was 6.1 and 8 was 6.2 (8.1 was 6.3). They finally bumped up the version to 10 with Windows 10.
    - twic 8y ago
      Java went from 1.4 to 5. Although not completely; it took until 9 for the actual directory names etc to change.
- ape4 8y ago
  Hey they changed some exit codes ;)
stevebmark 8y ago
Ripgrep is so awesome, "some performance improvements" doesn't make any sense because it's already horrifically fast.
heyoni 8y ago
goim was insanely good so I guess I’ll have to check this out!
- burntsushi 8y ago
  If you liked goim, then you'll definitely want to check out its replacement: https://github.com/BurntSushi/imdb-rename/ https://github.com/BurntSushi/imdb-rename/ It's even faster and comes with its own built in evaluation. There's also no more reliance on an external database.
m0zg 8y ago
The only reason why I install Rust on all my machines is to install ripgrep with Cargo. :-)
- mikojan 8y ago
  This.
cjohansson 8y ago
I love ripgrep and I’m thankful for the efforts that went into the project. It’s seems like a delicate programming project. Works great with Emacs btw
shaklee3 8y ago
Does anyone know how ripgrep's pcre support compares to python/Perl/awk/sed for speed? I noticed it can do substitutions, so I'm curious.
- asicsp 8y ago
  I haven't tried it yet, but it would depend on what sort of features you need Python wouldn't be a good choice for cli usage Perl is awesome to use from cli, and it is not just simple search and replace, see my tutorial[1] if you want to see examples sed and awk are awesome on their own for cli usage, sed is meant for line oriented tasks and awk for field oriented ones (there is overlap too) - one main difference compared to perl is that their regex is BRE/ERE which is generally faster but lacks in many features like lookarounds, non-greedy, named capture groups, etc you could check out sd[2] for a Rust implementation of sed like search and replacement (small subset of sed features) [1] https://github.com/learnbyexample/Command-line-text-processing/blob/master/perl_the_swiss_knife.md https://github.com/learnbyexample/Command-line-text-processi... [2] https://github.com/chmln/sd https://github.com/chmln/sd
  - shaklee3 8y ago
    Thanks! I was mostly asking because I've found Python's regular expressions to be extremely slow compared to egrep, and Perl is in the middle. It's annoying to have to call subprocess functions in python just too launch egrep on large files.
- burntsushi 8y ago
  ripgrep enables PCRE2's JIT, so it should be very fast.
zlynx 8y ago
I also love ripgrep and use it all the time. Thanks! I've lately been doing some Windows programming. Ripgrep from a Git Bash command-line can generally find ALL instances of some word before Visual Studio's Find In Solution can even return one result! The only downside is that rg does not understand namespaces when searching, so a commonly used member variable will show up all over the place. Can still usually find the right location by visual inspection before VS2017 though!
VohuMana 8y ago
I use this program on a daily basis at work, it is so fast and easy to use, I love it. Thank you for creating it :D
nindalf 8y ago
I’ve learned a lot by reading burntsushis code, so thank you for everything you’ve written. Separately, ripgrep is a great advertisement of what Rust is capable of. First, it shows it’s possible to write highly performant and reliable applications without resorting to non readable code. Second, it shows how simple it is to separate applications into a library + binary thanks to Rust’s package management. Ripgrep is merely a command line interface to libripgrep, which is reused in other applications like VS Code. The regex and directory walking code that was once part of the main ripgrep code base are now crates that are reused to great effect throughout the ecosystem.
- ngrilly 8y ago
  Last time I checked, VS Code didn't use librigpreg directly. It was calling the ripgrep binary instead.
  - burntsushi 8y ago
    This is correct. VS Code uses ripgrep's JSON output format from the binary. libripgrep is nice for building your own Rust programs for searching stuff. But there is a not insignificant amount if code that translates your argv into uses of libripgrep.
nudpiedo 8y ago
I cannot understand all that amount of excitement. Are we under heavy fanbot attack or am I missing something? If the whole thing is that, is faster than grep and "in Rust we Trust" then please feel free to ignore my comment. I would use OSX's slow grep every single day to find things and I never had any of the non-problems problems which ripgrep is trying to solve. About recursive, case insensitive and search and file number just use grep with the flags "-Rni", so I don't get in which aspect is superior. I never ever had half a thought about grep's performance at the systems I use it (already faster than my cognitive bottle neck). If I ever would need something better than grep I would try to move to the next level or deal with lots of unstructured data I would perhaps move to a local search engine using indexes. And personally I would consider an antipattern to skip the gitignore contents per default. What kind of machines or use cases are hitting the people in the comments?
- blux 8y ago
  Use case: grepping (big) repos. Rg is great for that, and grep, because it is much slower, almost not usable.
  - nudpiedo 8y ago
    maybe I did not had this problem yet... if I used grep on linux sources I would just skip the build folder and it usually works flawless (cannot even recall when was it too slow unless grepping over binary data). Do you refer maybe to deep interdependent javascript projects in which a small library requires hundreds of MB in dependencies? How many Gb have your projects and how many lines? Please include deps & builds (my current project is a small python one, 1.8Gb with deps, and I would usually grep only over the same folders all the time): $ time find ./ -type f -print|wc -l 140197 real 0m0.626s user 0m0.165s sys 0m0.473s Anyway, if it is about grepping over several gigabytes of unstructured code and there is no chance to have a better tool to index everything that could be a use case.
- Shoop 8y ago
  I use rg on a very large project (8.5 million LOC, 5.2G of files including the build). Searching with rg takes less than 2 seconds and I don't ever have to think about it. rg automatically handles not searching the build files and autogenerated files which often contain lots of matches but are never what I want to search. Additionally, rg has an excellent --vimgrep feature which makes it easy to integrate with the vim quickfix list. Vanilla grep, meanwhile takes 30 seconds -- too long to search while maintaining flow. For me, the whole thing _is_ that it is faster than grep. AFAIK, a lot of its speed is due to breaking compatibility with grep to allow for more convenient defaults (skip binary files by default, skip .gitignore by default, etc) which also happen to be faster. For me this is a clear win-win. Also, I have an 8 core machine. To be waiting for a 30s search knowing 7 cores are doing nothing makes me sad.
1023bytes 8y ago
What makes it so fast? Apart from skipping gitignored files and binaries. Parallelism? Some Rust-specific features?
- PudgePacket 8y ago
  https://github.com/BurntSushi/ripgrep#is-it-really-faster-than-everything-else https://github.com/BurntSushi/ripgrep#is-it-really-faster-th...
- Shoop 8y ago
  Here's a blog post from the author explaining how rg works: [1]. It's from 2016 so the benchmarks may not be up to date, including for rg's competitors. [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- deleted 8y ago
  [deleted]
stkrzysiak 8y ago
Here I thought I was hip using ag, the announcement blog post linked in this thread is fascinating and thorough, looks like it's time for me to try out rg.
linsomniac 8y ago
I tried ripgrep 0.10 when it came out. I did variations of "rg foo", "rg -i foo", "rg -ri foo", "rg -ri foo .". All of them returned 0 results. Where "ack foo | wc" returned 79 lines. Sooooo. I uninstalled "rg". I just tried again and same thing. If I run with "-u" I get results. I've just spent some time hunting and figured out that in my "projects" directory I had a ".gitignore" that had "", which I did a long time ago to stop "git status" from spamming me if I ran it in the wrong directory (base projects, a hg project which I have a lot of). For some reason "ack" recognizes that it is in a hg repo when I run it, and "rg" also recognizes that, but it ALSO* searches up the tree looking for a ".gitignore" which ack does not. Good to know.
- burntsushi 8y ago
  If you use the `--debug` flag, it will tell you which files are being ignored and why. If ripgrep doesn't search any files at all, then it should print a warning to stdout. ripgrep ascends your directory hierarchy for .gitignore files because that's how git works; .gitignore files in parent directories apply. ripgrep doesn't know anything about Mercurial, so it doesn't know to stop ascending your directories. ack is different since it doesn't look at your .gitignore files, so it doesn't have the same behavior here.
  - linsomniac 8y ago
    Oooh, ack ignores ".git" and ".hg", but ignores the ignore files. That, actually, seem like pretty ideal functionality. I guess the equivalent would be "rg --no-ignore-vcs". I had tried "-vvv" to no avail, thanks for the pointer about "--debug".
SanchoPanda 8y ago
The -pre flag is rarely mentioned but was a revelation for me in for dealing with MS office files and pdf documents. I use it every single day. alias rgpre='rg -i --max-columns=1500 --pre rgpre' *edit Adding my script here, and would be grateful for any improvements. Goal was to catch doc,docx,ppt,pptx,xls,xlsx, epub, mobi, and pdf and still work on termux with default packages there only. https://gist.github.com/ColonolBuendia/314826e37ec35c616d70506c38dc65aa https://gist.github.com/ColonolBuendia/314826e37ec35c616d705...
- ronjouch 8y ago
  That's indeed a lovely script. It makes for seamless searching my trove of PDFs and ebooks. Thanks for sharing it, and thanks BurntSushi for considering including it to rg's README :) !
- burntsushi 8y ago
  That's a lovely script. And yeah, I should try to publicize --pre a bit more, since it is pretty handy. I don't use it a ton myself, but when it works, I'm really thankful for it. (That might sound weird since it's in my tool, but I was initially kind of opposed to adding it.) One bummer is that it can slow ripgrep down quite a bit: one process per file is not good. As a tip, if you add --pre-glob '*.{pdf,xlsx,docx,doc,pptx,...}' and whatever else you need, then ripgrep will only invoke an extra process for those files specifically, which can significantly speed up ripgrep if you're searching a mixed directory (or just want to have `--pre` on by default).
  - SanchoPanda 8y ago
    That addition was an incredible time savings in my average search, I'm very grateful. A more prominent mention, and a sample script, would be very useful imho. Just googling for a way to do this for even just one of these file types yields inferior commercial solutions at best, often just nothing. And while I'm not sure how to make it much faster from a search perspective, I was able to find a 40% improvement in invocation by switching from rgpre to rgg as my alais. I ended here: alias rgg='rg -i --pre-glob '*.{pdf,xlsx,xls,docx,doc,pptx,ppt,html,epub,mobi}' --pre rgpre'

410 points by burntsushi 8y ago

Is there a way to output the results in Visual Studio format, which is <filename>(<line number>)<result>? This way when I hook up ripgrep as an External Tool, the line shown in the Output window can be clicked on to open the file and go to the specific line. I see in the source code that the separator is ":"; VS expects () around the line number.

burntsushi 8y ago
Not natively, no. You could probably write a simple script that transforms the output into what you want though. For example, on Unix at least, rg -n pattern | sed 's/:/(/' | sed 's/:/)/' That won't work in every case (e.g., in the case of a file path containining a `:`), but it might get you pretty far. And it could probably be made more precise. Otherwise, you could always use ripgrep's --json output mode to craft its output into whatever format you like, but that's probably more work. :-) The JSON format is documented here: https://docs.rs/grep-printer/0.1.2/grep_printer/struct.JSON.html https://docs.rs/grep-printer/0.1.2/grep_printer/struct.JSON....

j0e1 8y ago

I love ripgrep. For me it has done an exceptional job at searching and finding a large list of search strings over pretty large text files. Tangentially, it is because of great tools like these written in Rust that I have major respect for the language.

dfabulich 8y ago

If you've been holding back on Ripgrep because of its 0.x version numbering, now's the time to adopt it. Ripgrep is faster than grep, faster than ag (the Silver Searcher), and even faster than git grep. The author breaks down all of these details in this 17,000 word blog post, full of detailed benchmarks. https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ (It's from 2016, but it's still mostly in good shape; ripgrep has only gotten faster since then!)

loeg 8y ago
The blog post is good, and fair at the time. One caution I would add is that time has not stopped for the other projects, either. I believe ag (the silver searcher) has also seen considerable performance improvements since 2016.
- burntsushi 8y ago
  > I believe ag (the silver searcher) has also seen considerable performance improvements since 2016. Do you know what they are? I haven't noticed them, although it has been a while since I've rigorously benchmarked it. From looking at the source code, I don't think much has changed. e.g., it appears to still be using memory maps on Linux when searching large directories.
  - BeeOnRope 8y ago
    Out of curiosity, is there a better approach than memory maps?
    - shaklee3 8y ago
      The author goes over that here: https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
    - hungnmsu 8y ago
      Using mmap is generally slower than reading chunk of data and mmap does not work well if you have to handle huge files with several GBs of text data. You can find performance benchmark results and a detail analysis from @burntsushi blog and Lemire's blog post (https://lemire.me/blog/2012/06/26/which-is-fastest-read-fread-ifstream-or-mmap/ https://lemire.me/blog/2012/06/26/which-is-fastest-read-frea...).
- hungnmsu 8y ago
  ag is slower than ripgrep in all of my benchmarks (https://github.com/hungptit/fastgrep https://github.com/hungptit/fastgrep). And ag might be slower than GNU grep if files are stored in fast storage devices i.e SSD drives even though GNU grep is a single thread command.

sqs 8y ago

Ripgrep is awesome. Thank you for making it! In addition to being really fast, it "just works". By that I mean it automatically excludes the files I want to be excluded, like `.gitignore`d files and binary files. I know I can configure ack-grep (ag) and other tools to do that, but not needing to configure it is nice. BTW, if anyone hasn't read https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/, highly recommended. It's about how ripgrep is so fast. (Edit: Just saw another comment mentioned this, too. Goes to show that making a single high-quality blog post has a big impact.)

glangdale 8y ago
It is a very good blog post. Recommended! Now, I will do my obligatory "burntsushi needs to clag the rest of the Teddy code" post which I must do on every discussion of ripgrep. :-) The "subtitles_alternate_casei" examples would be a good benchmark - these 48 strings should not overwhelm Teddy as they could be sensibly merged into Teddy's 8 "buckets" (in fact, they could be merged into 5 buckets) using the simple greedy merging strategy in the original Teddy implementation. This would probably be a quite good project for someone who wants to contribute to ripgrep and could likely get some nice performance wins...
- burntsushi 8y ago
  I agree, this could be a nice project! If anyone wants to work on it, this is the place to start: https://github.com/rust-lang/regex/blob/master/src/literal/teddy_ssse3/imp.rs https://github.com/rust-lang/regex/blob/master/src/literal/t... I'll get to it myself eventually if someone else doesn't, but it will likely be a while.
  - glangdale 8y ago
    I've been thinking about a Teddy successor (which I suppose needs to be called "Taft"). Teddy is very much 'of its time' (SSSE3) and there are a lot of new approaches that seem interesting (AVX512 of Skylake generation, VBMI, Sunny Cove's even bigger slate of instructions, ARM NEON, SVE). I also have better ideas about followup confirm than I used to. There are also some prospect to pick a 'fragment' out of the whole string within Teddy or equivalent at a position not strictly at its suffix - this can even be done with ordering preserved if you are careful not to make fragment choices that allow o-o-o matches (only possible if strings overlap). I might do a bit of work on this, but I'm a bit jaded on string matching and regex matching after 13 years.

nojvek 8y ago

Vscode integrates ripgrep by default in their “search project” feature. I mostly use Vscode because of its fast search, decent auto complete and simple editing experience. Whoever made ripgrep, deserves massive kudos.

fiddlerwoaroof 8y ago
The OP is by the author of ripgrep
carlmr 8y ago
Same reason for switching for me. The projects at my work needed 2 minutes for a single regex search, with ripgrep it went to 3 seconds.
aashcan 8y ago
Atom's fuzzy finder also uses ripgrep as of 1.37.0 [0] https://github.com/atom/fuzzy-finder/pull/369 https://github.com/atom/fuzzy-finder/pull/369

keithnz 8y ago

version 0.10.0 to 11.0.0 is a big version leap!

ksherlock 8y ago
emacs did that too - 1.12 to 13.
- cpeterso 8y ago
  Solaris jumped from 2.6 to 7 (the "2." is now implicit). Of course, Windows NT jumped from 4.0 to 2000. ;)
  - LeonidasXIV 8y ago
    But it didn't, really. NT 4.0 was followed by NT 5.0 (branded Windows 2000) and then by 5.1 (Windows XP) and then by 6.0 (Windows 7).
    - signal11 8y ago
      Vista was 6.0. Windows 7 was 6.1 and 8 was 6.2 (8.1 was 6.3). They finally bumped up the version to 10 with Windows 10.
  - twic 8y ago
    Java went from 1.4 to 5. Although not completely; it took until 9 for the actual directory names etc to change.
ape4 8y ago
Hey they changed some exit codes ;)

stevebmark 8y ago

Ripgrep is so awesome, "some performance improvements" doesn't make any sense because it's already horrifically fast.

heyoni 8y ago

goim was insanely good so I guess I’ll have to check this out!

burntsushi 8y ago
If you liked goim, then you'll definitely want to check out its replacement: https://github.com/BurntSushi/imdb-rename/ https://github.com/BurntSushi/imdb-rename/ It's even faster and comes with its own built in evaluation. There's also no more reliance on an external database.

m0zg 8y ago

The only reason why I install Rust on all my machines is to install ripgrep with Cargo. :-)

mikojan 8y ago
This.

cjohansson 8y ago

I love ripgrep and I’m thankful for the efforts that went into the project. It’s seems like a delicate programming project. Works great with Emacs btw

shaklee3 8y ago

Does anyone know how ripgrep's pcre support compares to python/Perl/awk/sed for speed? I noticed it can do substitutions, so I'm curious.

asicsp 8y ago
I haven't tried it yet, but it would depend on what sort of features you need Python wouldn't be a good choice for cli usage Perl is awesome to use from cli, and it is not just simple search and replace, see my tutorial[1] if you want to see examples sed and awk are awesome on their own for cli usage, sed is meant for line oriented tasks and awk for field oriented ones (there is overlap too) - one main difference compared to perl is that their regex is BRE/ERE which is generally faster but lacks in many features like lookarounds, non-greedy, named capture groups, etc you could check out sd[2] for a Rust implementation of sed like search and replacement (small subset of sed features) [1] https://github.com/learnbyexample/Command-line-text-processing/blob/master/perl_the_swiss_knife.md https://github.com/learnbyexample/Command-line-text-processi... [2] https://github.com/chmln/sd https://github.com/chmln/sd
- shaklee3 8y ago
  Thanks! I was mostly asking because I've found Python's regular expressions to be extremely slow compared to egrep, and Perl is in the middle. It's annoying to have to call subprocess functions in python just too launch egrep on large files.
burntsushi 8y ago
ripgrep enables PCRE2's JIT, so it should be very fast.

zlynx 8y ago

I also love ripgrep and use it all the time. Thanks! I've lately been doing some Windows programming. Ripgrep from a Git Bash command-line can generally find ALL instances of some word before Visual Studio's Find In Solution can even return one result! The only downside is that rg does not understand namespaces when searching, so a commonly used member variable will show up all over the place. Can still usually find the right location by visual inspection before VS2017 though!

VohuMana 8y ago

I use this program on a daily basis at work, it is so fast and easy to use, I love it. Thank you for creating it :D

nindalf 8y ago

I’ve learned a lot by reading burntsushis code, so thank you for everything you’ve written. Separately, ripgrep is a great advertisement of what Rust is capable of. First, it shows it’s possible to write highly performant and reliable applications without resorting to non readable code. Second, it shows how simple it is to separate applications into a library + binary thanks to Rust’s package management. Ripgrep is merely a command line interface to libripgrep, which is reused in other applications like VS Code. The regex and directory walking code that was once part of the main ripgrep code base are now crates that are reused to great effect throughout the ecosystem.

ngrilly 8y ago
Last time I checked, VS Code didn't use librigpreg directly. It was calling the ripgrep binary instead.
- burntsushi 8y ago
  This is correct. VS Code uses ripgrep's JSON output format from the binary. libripgrep is nice for building your own Rust programs for searching stuff. But there is a not insignificant amount if code that translates your argv into uses of libripgrep.

nudpiedo 8y ago

I cannot understand all that amount of excitement. Are we under heavy fanbot attack or am I missing something? If the whole thing is that, is faster than grep and "in Rust we Trust" then please feel free to ignore my comment. I would use OSX's slow grep every single day to find things and I never had any of the non-problems problems which ripgrep is trying to solve. About recursive, case insensitive and search and file number just use grep with the flags "-Rni", so I don't get in which aspect is superior. I never ever had half a thought about grep's performance at the systems I use it (already faster than my cognitive bottle neck). If I ever would need something better than grep I would try to move to the next level or deal with lots of unstructured data I would perhaps move to a local search engine using indexes. And personally I would consider an antipattern to skip the gitignore contents per default. What kind of machines or use cases are hitting the people in the comments?

blux 8y ago
Use case: grepping (big) repos. Rg is great for that, and grep, because it is much slower, almost not usable.
- nudpiedo 8y ago
  maybe I did not had this problem yet... if I used grep on linux sources I would just skip the build folder and it usually works flawless (cannot even recall when was it too slow unless grepping over binary data). Do you refer maybe to deep interdependent javascript projects in which a small library requires hundreds of MB in dependencies? How many Gb have your projects and how many lines? Please include deps & builds (my current project is a small python one, 1.8Gb with deps, and I would usually grep only over the same folders all the time): $ time find ./ -type f -print|wc -l 140197 real 0m0.626s user 0m0.165s sys 0m0.473s Anyway, if it is about grepping over several gigabytes of unstructured code and there is no chance to have a better tool to index everything that could be a use case.
Shoop 8y ago
I use rg on a very large project (8.5 million LOC, 5.2G of files including the build). Searching with rg takes less than 2 seconds and I don't ever have to think about it. rg automatically handles not searching the build files and autogenerated files which often contain lots of matches but are never what I want to search. Additionally, rg has an excellent --vimgrep feature which makes it easy to integrate with the vim quickfix list. Vanilla grep, meanwhile takes 30 seconds -- too long to search while maintaining flow. For me, the whole thing _is_ that it is faster than grep. AFAIK, a lot of its speed is due to breaking compatibility with grep to allow for more convenient defaults (skip binary files by default, skip .gitignore by default, etc) which also happen to be faster. For me this is a clear win-win. Also, I have an 8 core machine. To be waiting for a 30s search knowing 7 cores are doing nothing makes me sad.

1023bytes 8y ago

What makes it so fast? Apart from skipping gitignored files and binaries. Parallelism? Some Rust-specific features?

PudgePacket 8y ago
https://github.com/BurntSushi/ripgrep#is-it-really-faster-than-everything-else https://github.com/BurntSushi/ripgrep#is-it-really-faster-th...
Shoop 8y ago
Here's a blog post from the author explaining how rg works: [1]. It's from 2016 so the benchmarks may not be up to date, including for rg's competitors. [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
deleted 8y ago
[deleted]

stkrzysiak 8y ago

Here I thought I was hip using ag, the announcement blog post linked in this thread is fascinating and thorough, looks like it's time for me to try out rg.

linsomniac 8y ago

I tried ripgrep 0.10 when it came out. I did variations of "rg foo", "rg -i foo", "rg -ri foo", "rg -ri foo .". All of them returned 0 results. Where "ack foo | wc" returned 79 lines. Sooooo. I uninstalled "rg". I just tried again and same thing. If I run with "-u" I get results. I've just spent some time hunting and figured out that in my "projects" directory I had a ".gitignore" that had "", which I did a long time ago to stop "git status" from spamming me if I ran it in the wrong directory (base projects, a hg project which I have a lot of). For some reason "ack" recognizes that it is in a hg repo when I run it, and "rg" also recognizes that, but it ALSO* searches up the tree looking for a ".gitignore" which ack does not. Good to know.

burntsushi 8y ago
If you use the `--debug` flag, it will tell you which files are being ignored and why. If ripgrep doesn't search any files at all, then it should print a warning to stdout. ripgrep ascends your directory hierarchy for .gitignore files because that's how git works; .gitignore files in parent directories apply. ripgrep doesn't know anything about Mercurial, so it doesn't know to stop ascending your directories. ack is different since it doesn't look at your .gitignore files, so it doesn't have the same behavior here.
- linsomniac 8y ago
  Oooh, ack ignores ".git" and ".hg", but ignores the ignore files. That, actually, seem like pretty ideal functionality. I guess the equivalent would be "rg --no-ignore-vcs". I had tried "-vvv" to no avail, thanks for the pointer about "--debug".

SanchoPanda 8y ago

The -pre flag is rarely mentioned but was a revelation for me in for dealing with MS office files and pdf documents. I use it every single day. alias rgpre='rg -i --max-columns=1500 --pre rgpre' *edit Adding my script here, and would be grateful for any improvements. Goal was to catch doc,docx,ppt,pptx,xls,xlsx, epub, mobi, and pdf and still work on termux with default packages there only. https://gist.github.com/ColonolBuendia/314826e37ec35c616d70506c38dc65aa https://gist.github.com/ColonolBuendia/314826e37ec35c616d705...

ronjouch 8y ago
That's indeed a lovely script. It makes for seamless searching my trove of PDFs and ebooks. Thanks for sharing it, and thanks BurntSushi for considering including it to rg's README :) !
burntsushi 8y ago
That's a lovely script. And yeah, I should try to publicize --pre a bit more, since it is pretty handy. I don't use it a ton myself, but when it works, I'm really thankful for it. (That might sound weird since it's in my tool, but I was initially kind of opposed to adding it.) One bummer is that it can slow ripgrep down quite a bit: one process per file is not good. As a tip, if you add --pre-glob '*.{pdf,xlsx,docx,doc,pptx,...}' and whatever else you need, then ripgrep will only invoke an extra process for those files specifically, which can significantly speed up ripgrep if you're searching a mixed directory (or just want to have `--pre` on by default).
- SanchoPanda 8y ago
  That addition was an incredible time savings in my average search, I'm very grateful. A more prominent mention, and a sample script, would be very useful imho. Just googling for a way to do this for even just one of these file types yields inferior commercial solutions at best, often just nothing. And while I'm not sure how to make it much faster from a search perspective, I was able to find a 40% improvement in invocation by switching from rgpre to rgg as my alais. I ended here: alias rgg='rg -i --pre-glob '*.{pdf,xlsx,xls,docx,doc,pptx,ppt,html,epub,mobi}' --pre rgpre'