3 ms·
About grep -o. # stat -c %s file 6297285 # file file file: ASCII text, with very long lines (1545), with CRLF, LF line terminators Imagine fi
by 1vuio0pswjnm7 3y ago
About grep -o.
# stat -c %s file
6297285
# file file
file: ASCII text, with very long lines (1545), with CRLF, LF line terminators
Imagine file as a wall of text.
1. Printing byte offsets.
# time grep -ob string file
0.03user 0.08system 0:00.22elapsed 52%CPU (0avgtext+0avgdata 1104maxresident)k
0inputs+0outputs (0major+86minor)pagefaults 0swaps
# rg -V
ripgrep 13.0.0
# time rg -ob string file
0.10user 0.17system 0:01.11elapsed 25%CPU (0avgtext+0avgdata 7804maxresident)k
0inputs+0outputs (0major+559minor)pagefaults 0swaps
# time sh -c "echo -n string|od -An -tx1|sed 's>^>/>;s/ //g'|ired -n file"
0.03user 0.09system 0:00.18elapsed 67%CPU (0avgtext+0avgdata 720maxresident)k
0inputs+0outputs (0major+189minor)pagefaults 0swaps
2. Printing some "context" around the matched string. For example, add characters immediately preceding string.
Baseline.
# time grep -o string file
0.02user 0.07system 0:00.15elapsed 65%CPU (0avgtext+0avgdata 1068maxresident)k
0inputs+0outputs (0major+84minor)pagefaults 0swaps
Add one character.
# time grep -o .string file
0.21user 0.08system 0:00.36elapsed 83%CPU (0avgtext+0avgdata 1088maxresident)k
0inputs+0outputs (0major+87minor)pagefaults 0swaps
Add another character.
# time grep -o ..string file
0.29user 0.09system 0:00.46elapsed 82%CPU (0avgtext+0avgdata 1064maxresident)k
0inputs+0outputs (0major+88minor)pagefaults 0swaps
# time rg -o ..string file
0.13user 0.13system 0:00.90elapsed 28%CPU (0avgtext+0avgdata 9012maxresident)k
0inputs+0outputs (0major+574minor)pagefaults 0swaps
Yikes.
Now let's try ired. Another shell script. This one will print all occurences of string.
cat > 1.sh << eof
#!/bin/sh
# usage: echo string|1.sh file [blocksize] [seek]"
read x;
x=$(echo -n $x|xxd -p);
echo /$x \
|ired -n $1 \
|sed "s/.*/s&@s-${3-0}@X$2/" \
|tr @ '\12' \
|ired -q -i /dev/stdin $1 \
|sed 's/.*/w&0a/' \
|ired -n /dev/stdout
eof
Baseline.
# echo string|time sh 1.sh 6
0.11user 0.10system 0:00.17elapsed 127%CPU (0avgtext+0avgdata 772maxresident)k
0inputs+0outputs (0major+466minor)pagefaults 0swaps
Add one character before string.
# echo string|time sh 1.sh 7 1
0.12user 0.09system 0:00.16elapsed 131%CPU (0avgtext+0avgdata 740maxresident)k
0inputs+0outputs (0major+473minor)pagefaults 0swaps
Add another.
# echo string|time sh 1.sh 8 2
0.12user 0.11system 0:00.20elapsed 112%CPU (0avgtext+0avgdata 744maxresident)k
0inputs+0outputs (0major+461minor)pagefaults 0swaps
Perhaps grep or ripgrep might be slightly faster at printing byte offsets.
But ired is faster at printing matches with context. (NB. Context here means characters, not lines.)
Try using ripgrep to print offsets for ired.
#!/bin/sh
read x;
rg --no-mmap -ob $x $1 \
|cut -d: -f1 \
|sed "s/.*/s&@s-${3-0}@X$2/" \
|tr @ '\12' \
|ired -q -i /dev/stdin $1 \
|sed 's/.*/w&0a/' \
|ired -n /dev/stdout
# time sh -c "echo string|1.sh file 8 2"
0.11user 0.06system 0:00.18elapsed 101%CPU (0avgtext+0avgdata 5972maxresident)k
0inputs+0outputs (0major+905minor)pagefaults 0swaps
# stat -c %s /usr/bin/ired /usr/bin/grep /usr/bin/rg
37544
219248
5074800
- burntsushi 3y agoOK, so first of all, let's get one thing cleared up. What the heck is ired? It isn't in the Archlinux package repos. I found this[1], but it looks like an incomplete and abandoned project. It doesn't even have proper docs: $ ired -h ired [-qhnv] [-c cmd] [-i script] [-|file ..] $ ired --help $ So like, I don't even know what `ired -n` is doing. From what I can tell from your commands, it's searching for `string`, but you first need to convert it to a hexadecimal representation. But okay, let's also check the output between the commands and make sure they're the same. I used my own file: $ time grep -ob string 1-2048.txt 333305:string 333380:string 920494:string 5166701:string 5210094:string 6775219:string real 0.006 user 0.006 sys 0.000 maxmem 15 MB faults 0 $ time rg -ob string 1-2048.txt 13123:333305:string 13124:333380:string 33382:920494:string 159885:5166701:string 161059:5210094:string 211466:6775219:string real 0.003 user 0.000 sys 0.003 maxmem 15 MB faults 0 $ time sh -c "echo -n string|od -An -tx1|sed 's>^>/>;s/ //g'|ired -n 1-2048.txt" 0x515f9 0x51644 0xe0bae 0x4ed66d 0x4f7fee 0x6761b3 real 0.013 user 0.010 sys 0.004 maxmem 15 MB faults 0 Indeed, the hexadecimal offsets printed by ired line up with the offsets printed by grep and ripgrep. Notice also the timing. ired is slower here for me. OK, now let's do context: $ time grep -ob string 1-2048.txt [..snip..] real 0.006 user 0.006 sys 0.000 maxmem 16 MB faults 0 $ time grep -ob .string 1-2048.txt [..snip..] real 0.005 user 0.003 sys 0.003 maxmem 16 MB faults 0 $ time grep -ob ..string 1-2048.txt [..snip..] real 0.006 user 0.003 sys 0.003 maxmem 16 MB faults 0 $ time rg -ob string 1-2048.txt [..snip..] real 0.004 user 0.003 sys 0.000 maxmem 16 MB faults 0 $ time rg -ob .string 1-2048.txt [..snip..] real 0.004 user 0.000 sys 0.003 maxmem 16 MB faults 0 $ time rg -ob ..string 1-2048.txt [..snip..] real 0.004 user 0.004 sys 0.000 maxmem 16 MB faults 0 I don't see anything worth saying "yikes" about here. One possible explanation for the timing differences is that your search has a lot of search results. The match count is a crucial part of benchmarking, and you've made the same mistake as the ugrep author by omitting them. But okay, let me try a search with more hits. $ time rg -ob the 1-2048.txt | wc -l 60509 real 0.011 user 0.006 sys 0.006 maxmem 16 MB faults 0 $ time rg -ob .the 1-2048.txt | wc -l 60477 real 0.014 user 0.014 sys 0.000 maxmem 16 MB faults 0 $ time rg -ob ..the 1-2048.txt | wc -l 60359 real 0.014 user 0.014 sys 0.000 maxmem 16 MB faults 0 A little slower, but that's what you'd expect with the higher match frequency. Now let's try your script for 1.sh: $ echo the | time sh 1.sh 1-2048.txt 6 | wc -l 63304 real 0.048 user 0.072 sys 0.052 maxmem 16 MB faults 0 $ echo the | time sh 1.sh 1-2048.txt 7 1 | wc -l 63336 real 0.056 user 0.096 sys 0.042 maxmem 16 MB faults 0 $ echo the | time sh 1.sh 1-2048.txt 8 2 | wc -l 63419 real 0.053 user 0.079 sys 0.049 maxmem 16 MB faults 0 (The counts are a little different because `..the` matches fewer things than `the` when given to grep, but presumably `ired` doesn't care about that.) But in any case, ired is quite a bit slower here. OK, let's pop up a level. Your benchmark is somewhat flawed. For three reasons. First is because the timings are so short that the differences here are generally irrelevant to human perception. It reminds me of the time when ripgrep came out, and someone would respond with a "gotcha" that `ag` was faster because it ran a search on a tiny repository in 10ms where as ripgrep took 12ms. That's not quite exactly the same as what's happening here, but it's close. The second is that the haystack is so short that overhead is likely playing a role here. The timings are just too short to be reliable indicators of performance as the haystack size scales. See my commentary on ugrep's benchmarks[2]. Let's try a bigger file: $ stat -c %s eigth.txt 1621035918 $ file eigth.txt eigth.txt: ASCII text $ time rg -ob Sherlock eigth.txt | wc -l 1068 real 0.154 user 0.103 sys 0.050 maxmem 1551 MB faults 0 $ time rg -ob .Sherlock eigth.txt | wc -l 935 real 0.156 user 0.096 sys 0.060 maxmem 1551 MB faults 0 $ time rg -ob ..Sherlock eigth.txt | wc -l 932 real 0.154 user 0.107 sys 0.047 maxmem 1551 MB faults 0 And now ired: $ echo Sherlock | time sh 1.sh eigth.txt 6 | wc -l 1068 real 1.393 user 0.671 sys 0.729 maxmem 16 MB faults 0 $ echo Sherlock | time sh 1.sh eigth.txt 7 1 | wc -l 1201 real 1.391 user 0.604 sys 0.793 maxmem 16 MB faults 0 $ echo Sherlock | time sh 1.sh eigth.txt 8 2 | wc -l 1204 real 1.395 user 0.578 sys 0.823 maxmem 16 MB faults 0 Yikes. Over an order of magnitude slower. Now that the memory usage reported for ripgrep is high just because it's using file backed memory maps. It's not actual heap usage. You can check this by disabling memory maps: $ time rg -ob ..Sherlock eigth.txt --no-mmap | wc -l 932 real 0.179 user 0.063 sys 0.116 maxmem 16 MB faults 0 And if we increase the match frequency on the same large haystack, the gap closes a little, but ired is still about 4x slower: $ time rg -ob ..the eigth.txt | wc -l 13141187 real 2.470 user 2.418 sys 0.050 maxmem 1551 MB faults 0 $ echo the | time sh 1.sh eigth.txt 8 2 | wc -l 13894916 real 10.027 user 16.293 sys 8.122 maxmem 402 MB faults 0 I'm not clear on why you're seeing the results you are. It could be because your haystack is so small that you're mostly just measuring noise. ripgrep 14 did introduce some optimizations in workloads like this by reducing match overhead, but I don't think it's anything huge in this case. (And I just tried ripgrep 13 on the same commands above and the timings are similar if a tiny bit slower.) [1]: https://github.com/radare/ired https://github.com/radare/ired [2]: https://github.com/BurntSushi/ripgrep/discussions/2597 https://github.com/BurntSushi/ripgrep/discussions/2597