3 ms·

Yeah, rg is definitely not always the fastest. Test for your own use case and all. In pretty much any case involving more complicated queries and/or unicode I

by opencl 7y ago

Yeah, rg is definitely not always the fastest. Test for your own use case and all. In pretty much any case involving more complicated queries and/or unicode I have found rg to be the fastest by a very wide margin. Also if you're searching multiple files rg is multithreaded by default without having to pass through something like xargs or parallel. If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation.

burntsushi 7y ago
Thanks for noticing ripgrep's performance on Unicode. :-) > If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation. This is most definitely wrong. Firstly, ripgrep doesn't do any UTF-8 validation. That would make it much much slower. Secondly, searching a simple literal is one of the cases where ripgrep should actually do better than GNU grep in many cases. This is because of the slightly smarter way in which ripgrep keeps its vectorized loop active more often than GNU grep does by trying to search rarer bytes. I talk about this quite a bit here, which even includes a benchmark where I measured ripgrep at 2x the speed of GNU grep for a simple literal search: https://blog.burntsushi.net/ripgrep/#subtitles-literal https://blog.burntsushi.net/ripgrep/#subtitles-literal Anyone can try this for themselves: $ cd /tmp $ curl 'https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz' | gzip -cd > subtitles.en.txt $ pv < subtitles.en.txt > /dev/null 9.28GiB 0:00:01 [4.89GiB/s] [======================================================================================================================================================>] 100% $ time grep ZQZQZQZQ < subtitles.en.txt real 8.713 user 7.495 sys 1.212 maxmem 9 MB faults 0 $ time rg ZQZQZQZQ < subtitles.en.txt real 1.857 user 0.573 sys 1.282 maxmem 9 MB faults 0 On my system, /tmp is a ramdisk, so this doesn't factor in I/O. If you look up this thread, you'll note that someone was trying to search a 30GB file. Unless that person could fit that file into memory, it's very likely that they were just measuring the I/O bandwidth of their machine.
- opencl 7y ago
  Thanks for the detailed explanation and your work on rg, I use it all the time and strongly appreciate the performance with unicode because I do a lot of searching of large directories of files that are largely in Japanese.
  - burntsushi 7y ago
    No problem! If it's not too much trouble and you can share your data (and queries), I'm always happy to see more details about use cases that take advantage of ripgrep's Unicode support and its associated performance characteristics. At some point, I'm going to have to redo my previously published benchmarks, and it would be great to get some real use cases like yours in there. :-)
    - opencl 7y ago
      A lot of the data I work on is unfortunately NDAed but I'll try to find something that can be shared, no guarantees though. What's a good way to contact you with this stuff?
      - burntsushi 7y ago
        Thank you! Just file an issue and that should be good enough!

by opencl 7y ago

burntsushi 7y ago

Thanks for noticing ripgrep's performance on Unicode. :-) > If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation. This is most definitely wrong. Firstly, ripgrep doesn't do any UTF-8 validation. That would make it much much slower. Secondly, searching a simple literal is one of the cases where ripgrep should actually do better than GNU grep in many cases. This is because of the slightly smarter way in which ripgrep keeps its vectorized loop active more often than GNU grep does by trying to search rarer bytes. I talk about this quite a bit here, which even includes a benchmark where I measured ripgrep at 2x the speed of GNU grep for a simple literal search: https://blog.burntsushi.net/ripgrep/#subtitles-literal https://blog.burntsushi.net/ripgrep/#subtitles-literal Anyone can try this for themselves: $ cd /tmp $ curl 'https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz' | gzip -cd > subtitles.en.txt $ pv < subtitles.en.txt > /dev/null 9.28GiB 0:00:01 [4.89GiB/s] [======================================================================================================================================================>] 100% $ time grep ZQZQZQZQ < subtitles.en.txt real 8.713 user 7.495 sys 1.212 maxmem 9 MB faults 0 $ time rg ZQZQZQZQ < subtitles.en.txt real 1.857 user 0.573 sys 1.282 maxmem 9 MB faults 0 On my system, /tmp is a ramdisk, so this doesn't factor in I/O. If you look up this thread, you'll note that someone was trying to search a 30GB file. Unless that person could fit that file into memory, it's very likely that they were just measuring the I/O bandwidth of their machine.

opencl 7y ago
Thanks for the detailed explanation and your work on rg, I use it all the time and strongly appreciate the performance with unicode because I do a lot of searching of large directories of files that are largely in Japanese.
- burntsushi 7y ago
  No problem! If it's not too much trouble and you can share your data (and queries), I'm always happy to see more details about use cases that take advantage of ripgrep's Unicode support and its associated performance characteristics. At some point, I'm going to have to redo my previously published benchmarks, and it would be great to get some real use cases like yours in there. :-)
  - opencl 7y ago
    A lot of the data I work on is unfortunately NDAed but I'll try to find something that can be shared, no guarantees though. What's a good way to contact you with this stuff?
    - burntsushi 7y ago
      Thank you! Just file an issue and that should be good enough!