3 ms·

grep is a general purpose tool for searching for text in all types of files, baked into the standards for UNIX. Some programmers use it to search source code. O

grep is a general purpose tool for searching for text in all types of files, baked into the standards for UNIX. Some programmers use it to search source code. Other people use it for other types of text searches that have nothing to do with source code, they rely on it in scripts, they don't use it as part of a text-based programmer UI, they rely on it to never crash, etc. ripgrep is a specialist, opinionated tool, designed primarily to search through source code repositories. There's not much you can add to general purpose text search to make it faster; you can make it use mmap() at the risk of it crashing on truncated files, you can reduce the expressiveness of regular expressions so they can be computed faster. You could throw out general support for all locales and charsets and hardcode support for only UTF-8 / UTF-16, but you shouldn't.

bawolff 3y ago
> you can make it use mmap() at the risk of it crashing on truncated files I was under the impression that grep removed mmap() support because it was slower than normal file i/o
- burntsushi 3y ago
  I talked about this in the OP. Memory maps are sometimes a little faster. See: $ ls -l full.txt -rw-rw-r-- 1 andrew users 13113340782 Sep 29 12:30 full.txt $ time rg -c --no-mmap Clipton full.txt 294 real 1.337 user 0.470 sys 0.866 maxmem 15 MB faults 0 $ time rg -c --mmap Clipton full.txt 294 real 1.045 user 0.722 sys 0.323 maxmem 12511 MB faults 0 But in recursive search, especially when used for lots of little files, they end up provoking substantial overhead that slows everything down. And this might change depending on the platform.
burntsushi 3y ago
> There's not much you can add to general purpose text search to make it faster Oh I beg to differ! The blog post goes into this. Here's a simple demonstration using ripgrep 14: $ ls -l full.txt -rw-rw-r-- 1 andrew users 13113340782 Sep 29 12:30 full.txt $ time rg -c --no-mmap 'Clipton' full.txt 294 real 1.419 user 0.539 sys 0.879 maxmem 15 MB faults 0 $ time LC_ALL=C grep -c 'Clipton' full.txt 294 real 6.911 user 6.078 sys 0.829 maxmem 15 MB faults 0 $ time rg -c --no-mmap 'DMZ|Clipton' full.txt 1070 real 1.643 user 0.747 sys 0.894 maxmem 15 MB faults 0 $ time LC_ALL=C grep -E -c 'DMZ|Clipton' full.txt 1070 real 8.317 user 7.384 sys 0.930 maxmem 15 MB faults 0 No memory maps. No multi-threading. No filtering. No fancy regex engine features or reducing expressiveness. No locales. No UTF-8. No UTF-16. Just a simple literal and a simple alternation of literals. It's just better algorithms. Also, you can disable ripgrep's opinions with `-uuu`. It's not designed to just be for code searching. You can use it for normal grepping too. It will even automatically revert to the standard grep line format in shell pipelines.

bawolff 3y ago

> you can make it use mmap() at the risk of it crashing on truncated files I was under the impression that grep removed mmap() support because it was slower than normal file i/o

burntsushi 3y ago
I talked about this in the OP. Memory maps are sometimes a little faster. See: $ ls -l full.txt -rw-rw-r-- 1 andrew users 13113340782 Sep 29 12:30 full.txt $ time rg -c --no-mmap Clipton full.txt 294 real 1.337 user 0.470 sys 0.866 maxmem 15 MB faults 0 $ time rg -c --mmap Clipton full.txt 294 real 1.045 user 0.722 sys 0.323 maxmem 12511 MB faults 0 But in recursive search, especially when used for lots of little files, they end up provoking substantial overhead that slows everything down. And this might change depending on the platform.

burntsushi 3y ago

> There's not much you can add to general purpose text search to make it faster Oh I beg to differ! The blog post goes into this. Here's a simple demonstration using ripgrep 14: $ ls -l full.txt -rw-rw-r-- 1 andrew users 13113340782 Sep 29 12:30 full.txt $ time rg -c --no-mmap 'Clipton' full.txt 294 real 1.419 user 0.539 sys 0.879 maxmem 15 MB faults 0 $ time LC_ALL=C grep -c 'Clipton' full.txt 294 real 6.911 user 6.078 sys 0.829 maxmem 15 MB faults 0 $ time rg -c --no-mmap 'DMZ|Clipton' full.txt 1070 real 1.643 user 0.747 sys 0.894 maxmem 15 MB faults 0 $ time LC_ALL=C grep -E -c 'DMZ|Clipton' full.txt 1070 real 8.317 user 7.384 sys 0.930 maxmem 15 MB faults 0 No memory maps. No multi-threading. No filtering. No fancy regex engine features or reducing expressiveness. No locales. No UTF-8. No UTF-16. Just a simple literal and a simple alternation of literals. It's just better algorithms. Also, you can disable ripgrep's opinions with `-uuu`. It's not designed to just be for code searching. You can use it for normal grepping too. It will even automatically revert to the standard grep line format in shell pipelines.