5 ms·

Ripgrep 0.10.0 released (PCRE2 and multi-line support)

180 points by burntsushi 8y ago

eridius 8y ago
Looks great! I'm curious what the performance implications of switching to PCRE2 are.
- burntsushi 8y ago
  PCRE2 is opt-in. For performance, here's a start. :) https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcre2-slow https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcr...
  - eridius 8y ago
    Ah. I recommend adding a link to that FAQ entry in the README, as it currently mentions in several places that it has optional PCRE support but has zero mention of the fact that this is (usually) slower.
  - dagenix 8y ago
    My extremely unscientific test had PCRE2 being quite a bit slower - 1 second vs 0.2 seconds to search through 10k files - ie, slower, but still pretty darn fast.
    - burntsushi 8y ago
      That feels like it's in about the right ballpark based on my own tests. You can make it go a bit faster if you're using nightly Rust and compile with SIMD optimizations: RUSTFLAGS="-C target-cpu=native" cargo build --release --features 'simd-accel avx-accel pcre2' In particular, this enables SIMD optimizations in the crate ripgrep uses for decoding UTF-8, which is required to run on all files in order to use PCRE2 in Unicode mode. (Eventually, I'm hoping the SIMD optimizations in the encoding_rs crate move to using runtime detection like the regex crate instead of requiring compile time flags.) Or alternatively, disable Unicode with `--no-pcre2-unicode` and no transcoding will be done. :-)
      - dagenix 8y ago
        I installed with: PCRE2_SYS_STATIC=1 RUSTFLAGS="-C target-cpu=native" rustup run nightly cargo install ripgrep --force --features 'simd-accel avx-accel pcre2' `--no-pcre2-unicode` sped it up a little. But, it was already quite fast. Great work on ripgrep and thanks for such an amazing tool!
anyfoo 8y ago
I've been using ag, the silver searcher, for a few years now. It's blazingly fast (easily searching a massive code base in single-digit seconds), automatically ignores files I almost never want to be searched (anything in .git for example), and has convenient defaults around color highlighting and showing context. I'm more than open to add ripgrep to my collection of everyday tools, however, if there is a real benefit. Does ripgrep do anything that would replace or complement my ag usage? Note that it may also be possible that for new users, ripgrep is the better tool, but that ag is still more than good enough if you already based workflows and habits around it.
- andy128k 8y ago
  I also use ag for years. Recently I tried to use rg and found it is even faster than ag. Few times I compared results and somehow ag skipped few files but rg found matches. My reflexes are still to type ag but I already want to switch to rg.
  - meowface 8y ago
    alias ag="rg" is what I've just started doing. Both tools are great, but rg is just better and faster, while also not missing anything that ag has (at least from my own experience).
- dfabulich 8y ago
  riggrep is shockingly even faster than ag. (I couldn't believe it.) https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ > My hope is that this article not only convinced you that rg is quite fast, but more importantly, that you found my analysis of each benchmark educational. String searching is an old problem in computer science, but there is still plenty of work left to do to advance the state of the art.
- burntsushi 8y ago
  ripgrep does everything you say about ag, and has much less buggy support for gitignore. It is also quite a bit faster for bigger workloads. For smaller work loads you might not notice a difference. However, ripgrep is good enough to even replace uses of grep where ag might otherwise behave sub-optimally. Specifically, with this release, ag no longer has any major features that ripgrep doesn't have. Additionally, ripgrep gives you almost every flag found in GNU grep, and then some. I think some of this is addressed in the README. :)
danso 8y ago
`ack` has been my goto for grep-likes -- though I would occasionally `ag` for the multi-line support, which ack does not have. My favorite feature of ack that ag didn't have was the --output flag, which would let you do something like this: $ ack '^(\w+)' --output 'hello $1 world' hello abc world hello defgh world hello etc world But seems like this is equivalent to rg's `-or` flag, so looks like I don't have any reason to not switch over. I love the `-r` flag by itself (short for `--replace`), which seems like a handy replacement for sed's substitution. Also, I've been impressed with how responsive and receptive the author has been on Github. And it doesn't hurt that all of other burntsushi's tools seem solid and well-designed, such as xsv.
- icc97 8y ago
  You can use ripgrep with Ack. We'll at least you can with Vim [0], so I assume it works outside Vim too. [0]: https://github.com/mileszs/ack.vim https://github.com/mileszs/ack.vim
staticassertion 8y ago
One of the first things i install on a new system. I gave it to a friend who's just learning to program and has a hard time organizing their projects still - it's been an absolute lifesaver.
nothrows 8y ago
sticking with ack. a tool for humans with decent .ignore file support. ag and rg never seem to ignore the subdirectories I want.
- burntsushi 8y ago
  Can you give a specific example? I'd be happy to try and work though it with you! ripgrep certainly supports .ignore files. Last I checked, ack specifically doesn't support .ignore files: https://beyondgrep.com/feature-comparison/ https://beyondgrep.com/feature-comparison/

180 points by burntsushi 8y ago

eridius 8y ago

Looks great! I'm curious what the performance implications of switching to PCRE2 are.

burntsushi 8y ago
PCRE2 is opt-in. For performance, here's a start. :) https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcre2-slow https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcr...
- eridius 8y ago
  Ah. I recommend adding a link to that FAQ entry in the README, as it currently mentions in several places that it has optional PCRE support but has zero mention of the fact that this is (usually) slower.
- dagenix 8y ago
  My extremely unscientific test had PCRE2 being quite a bit slower - 1 second vs 0.2 seconds to search through 10k files - ie, slower, but still pretty darn fast.
  - burntsushi 8y ago
    That feels like it's in about the right ballpark based on my own tests. You can make it go a bit faster if you're using nightly Rust and compile with SIMD optimizations: RUSTFLAGS="-C target-cpu=native" cargo build --release --features 'simd-accel avx-accel pcre2' In particular, this enables SIMD optimizations in the crate ripgrep uses for decoding UTF-8, which is required to run on all files in order to use PCRE2 in Unicode mode. (Eventually, I'm hoping the SIMD optimizations in the encoding_rs crate move to using runtime detection like the regex crate instead of requiring compile time flags.) Or alternatively, disable Unicode with `--no-pcre2-unicode` and no transcoding will be done. :-)
    - dagenix 8y ago
      I installed with: PCRE2_SYS_STATIC=1 RUSTFLAGS="-C target-cpu=native" rustup run nightly cargo install ripgrep --force --features 'simd-accel avx-accel pcre2' `--no-pcre2-unicode` sped it up a little. But, it was already quite fast. Great work on ripgrep and thanks for such an amazing tool!

anyfoo 8y ago

I've been using ag, the silver searcher, for a few years now. It's blazingly fast (easily searching a massive code base in single-digit seconds), automatically ignores files I almost never want to be searched (anything in .git for example), and has convenient defaults around color highlighting and showing context. I'm more than open to add ripgrep to my collection of everyday tools, however, if there is a real benefit. Does ripgrep do anything that would replace or complement my ag usage? Note that it may also be possible that for new users, ripgrep is the better tool, but that ag is still more than good enough if you already based workflows and habits around it.

andy128k 8y ago
I also use ag for years. Recently I tried to use rg and found it is even faster than ag. Few times I compared results and somehow ag skipped few files but rg found matches. My reflexes are still to type ag but I already want to switch to rg.
- meowface 8y ago
  alias ag="rg" is what I've just started doing. Both tools are great, but rg is just better and faster, while also not missing anything that ag has (at least from my own experience).
dfabulich 8y ago
riggrep is shockingly even faster than ag. (I couldn't believe it.) https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ > My hope is that this article not only convinced you that rg is quite fast, but more importantly, that you found my analysis of each benchmark educational. String searching is an old problem in computer science, but there is still plenty of work left to do to advance the state of the art.
burntsushi 8y ago
ripgrep does everything you say about ag, and has much less buggy support for gitignore. It is also quite a bit faster for bigger workloads. For smaller work loads you might not notice a difference. However, ripgrep is good enough to even replace uses of grep where ag might otherwise behave sub-optimally. Specifically, with this release, ag no longer has any major features that ripgrep doesn't have. Additionally, ripgrep gives you almost every flag found in GNU grep, and then some. I think some of this is addressed in the README. :)

danso 8y ago

`ack` has been my goto for grep-likes -- though I would occasionally `ag` for the multi-line support, which ack does not have. My favorite feature of ack that ag didn't have was the --output flag, which would let you do something like this: $ ack '^(\w+)' --output 'hello $1 world' hello abc world hello defgh world hello etc world But seems like this is equivalent to rg's `-or` flag, so looks like I don't have any reason to not switch over. I love the `-r` flag by itself (short for `--replace`), which seems like a handy replacement for sed's substitution. Also, I've been impressed with how responsive and receptive the author has been on Github. And it doesn't hurt that all of other burntsushi's tools seem solid and well-designed, such as xsv.

icc97 8y ago
You can use ripgrep with Ack. We'll at least you can with Vim [0], so I assume it works outside Vim too. [0]: https://github.com/mileszs/ack.vim https://github.com/mileszs/ack.vim

staticassertion 8y ago

One of the first things i install on a new system. I gave it to a friend who's just learning to program and has a hard time organizing their projects still - it's been an absolute lifesaver.

nothrows 8y ago

sticking with ack. a tool for humans with decent .ignore file support. ag and rg never seem to ignore the subdirectories I want.

burntsushi 8y ago
Can you give a specific example? I'd be happy to try and work though it with you! ripgrep certainly supports .ignore files. Last I checked, ack specifically doesn't support .ignore files: https://beyondgrep.com/feature-comparison/ https://beyondgrep.com/feature-comparison/