3 ms·
> rg uses a lot of memory in the OpenSubtitles test. 903M vs 29M for ugrep. Unlike the previous test, we are not told the size of the file being searched. Whic
by burntsushi 3y ago
> rg uses a lot of memory in the OpenSubtitles test. 903M vs 29M for ugrep. Unlike the previous test, we are not told the size of the file being searched.
Which test exactly? That's just likely because of memory maps futzing with the RSS data. Not actually more heap memory. Try with --no-mmap.
I'm not sure I understand the rest of your comment about grep -o. Grep tools usually have a flag to print the offset of each match.
EDIT: Now that I have hands on a keyboard, I'll demonstrate the mmap thing. First, ugrep:
$ time ugrep-4.4.1 -c '\w+\s+Sherlock\s+Holmes\s+\w+' sixteenth.txt
72
real 22.115
user 22.015
sys 0.093
maxmem 30 MB
faults 0
$ time ugrep-4.4.1 -c '\w+\s+Sherlock\s+Holmes\s+\w+' sixteenth.txt --mmap
72
real 21.776
user 21.749
sys 0.020
maxmem 802 MB
faults 0
And now for ripgrep:
$ time rg-14.0.3 -c '\w+\s+Sherlock\s+Holmes\s+\w+' opensubtitles/2018/en/sixteenth.txt
72
real 0.076
user 0.046
sys 0.030
maxmem 779 MB
faults 0
$ time rg-14.0.3 -c '\w+\s+Sherlock\s+Holmes\s+\w+' opensubtitles/2018/en/sixteenth.txt --no-mmap
72
real 0.087
user 0.033
sys 0.053
maxmem 15 MB
faults 0
It looks like the difference here is that ripgrep chooses to use a memory map by default here. I don't think it makes much of a difference here.
If the file were bigger than available memory, then the OS would automatically handle paging.
- deleted 3y ago[deleted]