5 ms·
1. Retrieve test.json curl -i40A "" "https://api.crossref.org/works?query=unix&rows=1000" > test.json 2. Create shell script #!/bin/sh # usa
by 1vuio0pswjnm7 3y ago
1. Retrieve test.json
curl -i40A "" "https://api.crossref.org/works?query=unix&rows=1000" > test.json
2. Create shell script
#!/bin/sh
# usage: echo string|1.sh file [blocksize] [seek]"
read x;
x=$(echo -n $x|od -An -tx1|tr -d '\40');
echo /$x \
|ired -n $1 \
|sed "s/.*/s&@s-${3-0}@X$2/" \
|tr @ '\12' \
|ired -q -i /dev/stdin $1 \
|sed 's/.*/w&0a/' \
|ired -n /dev/stdout
We can make the script slightly faster by using busybox
#!/bin/sh
# usage: echo string|1.sh file [blocksize] [seek]"
read x;
x=$(echo -n $x|busybox od -An -tx1|busybox tr -d '\40');
echo /$x \
|ired -n $1 \
|busybox sed "s/.*/s&@s-${3-0}@X$2/" \
|busybox tr @ '\12' \
|ired -q -i /dev/stdin $1 \
|busybox sed 's/.*/w&0a/' \
|ired -n /dev/stdout
NB. If redirecting output to a file, replace /dev/stdout with the file name.
ired is available on Void Linux
https://ftp.lysator.liu.se/pub/voidlinux/static/ https://ftp.lysator.liu.se/pub/voidlinux/static/
xbps-query.static -Rs ired-0
xbps-install.static ired
3. Test grep v3.6, ripgrep v14.0.3 and shell script; busybox is v1.34.1
busybox time grep -Eo .{35}https:.{4} test.json;
busybox time rg -o .{35}https:.{4} test.json;
busybox time sh -c "echo https:|1.sh 45 35 test.json"
We can make the script slower by using bash
busybox time bash -c "echo https:|1.sh 45 35 test.json"
Program size
du -h /usr/bin/grep
216K/usr/bin/grep
du -h /usr/bin/rg
5.7M/usr/bin/rg
du -hc /usr/bin/ired /bin/dash /usr/bin/tr /usr/bin/sed /usr/bin/od
456K/bin/dash
40K/usr/bin/ired
56K/usr/bin/tr
68K/usr/bin/od
104K/usr/bin/sed
724Ktotal
du -h /usr/bin/busybox /usr/bin/ired
772K/usr/bin/busybox
40K/usr/bin/ired
812Ktotal
readelf -d /bin/dash /usr/bin/busybox
File: /bin/dash
There is no dynamic section in this file.
File: /usr/bin/busybox
There is no dynamic section in this file.
- burntsushi 3y agoOK, so I'll try your commands: $ busybox time grep -Eo .{35}https:.{4} test.json real 0m 0.15s user 0m 0.15s sys 0m 0.00s $ busybox time rg-14.0.3 -o .{35}https:.{4} test.json real 0m 0.00s user 0m 0.00s sys 0m 0.00s $ busybox time dash -c "echo https:|./1.sh test.json 45 35" real 0m 0.01s user 0m 0.01s sys 0m 0.00s $ busybox time bash -c "echo https:|./1.sh test.json 45 35" real 0m 0.00s user 0m 0.00s sys 0m 0.00s $ busybox time dash -c "echo https:|./busy-1.sh test.json 45 35" real 0m 0.00s user 0m 0.01s sys 0m 0.00s $ busybox time bash -c "echo https:|./busy-1.sh test.json 45 35" real 0m 0.01s user 0m 0.01s sys 0m 0.00s So grep -o takes 150ms, but both ripgrep and ired are seemingly instant. But if I use zsh's builtin `time` command with my own TIMEFMT[1], it gives me numbers greater than 0: $ time grep -Eo .{35}https:.{4} test.json real 0.324 user 0.317 sys 0.007 maxmem 16 MB faults 0 $ time rg-14.0.3 -o .{35}https:.{4} test.json real 0.008 user 0.003 sys 0.003 maxmem 16 MB faults 0 $ time dash -c "echo https:|./1.sh test.json 45 35" real 0.010 user 0.011 sys 0.007 maxmem 16 MB faults 0 $ time bash -c "echo https:|./1.sh test.json 45 35" real 0.011 user 0.014 sys 0.004 maxmem 16 MB faults 0 Would you look at that. ripgrep is faster! By a whole 2 milliseconds! WOW! OK, since I'm a software developer and thus apparently cannot understand the lowly needs of an "ordinary user," I'll hop over to my machine with a i5-7600, which was released 6 years ago. Is that ordinary enough, or still too super charged to do any meaningful comparison whatsoever? $ time grep -Eo .{35}https:.{4} test.json real 0.641 user 0.620 sys 0.017 maxmem 6 MB faults 0 $ time rg-14.0.3 -o .{35}https:.{4} test.json real 0.010 user 0.008 sys 0.000 maxmem 8 MB faults 0 $ time dash -c "echo https:|./1.sh test.json 45 35" real 0.011 user 0.009 sys 0.011 maxmem 6 MB faults 0 $ time bash -c "echo https:|./1.sh test.json 45 35" real 0.013 user 0.021 sys 0.003 maxmem 6 MB faults 0 (I ran the commands above each several times and took the minimum.) OK, so ripgrep is still 1ms faster even on "ordinary user" hardware. All right, so your other comment also shared another benchmark: $ time grep -Eo .{100}https:.{50} test.json real 1.777 user 1.772 sys 0.003 maxmem 6 MB faults 0 $ time rg-14.0.3 -o .{100}https:.{50} test.json real 0.013 user 0.006 sys 0.000 maxmem 8 MB faults 0 $ time rg-14.0.3 --color never -o .{100}https:.{50} test.json real 0.006 user 0.006 sys 0.000 maxmem 8 MB faults 0 $ time dash -c "echo https:|./1.sh test.json 156 100" real 0.015 user 0.024 sys 0.004 maxmem 7 MB faults 0 $ time bash -c "echo https:|./1.sh test.json 156 100" real 0.016 user 0.028 sys 0.000 maxmem 7 MB faults 0 (Notice that disabling color and line numbers for ripgrep improves its speed a fair bit. ired isn't doing either of those things, so it's only fair. GNU grep doesn't count line numbers by default and disabling color doesn't improve its perf here.) This one is more interesting because it exposes the fact that many regex engines have trouble dealing with bounded repeats. Something like `.{100}` for example is not executed particularly efficiently in most regex engines. And indeed, in ripgrep by default, `.` actually matches the UTF-8 encoding of any Unicode scalar value (so between 1 and 4 bytes) and not any arbitrary byte. You'd need to pass the `--no-unicode` flag or prefix your pattern with `(?-u)` to match any arbitrary byte. And indeed, even then, `.` doesn't match `\n`. So you might even want `(?s-u)`. But since this is a grep and *greps are line oriented*, you'd need to enable multi-line mode in ripgrep (GNU grep doesn't have this): $ time rg-14.0.3 -Uo '(?s-u).{100}https:.{50}' test.json real 0.057 user 0.041 sys 0.006 maxmem 8 MB faults 0 $ time rg-14.0.3 --color never -N -Uo '(?s-u).{100}https:.{50}' test.json real 0.042 user 0.041 sys 0.000 maxmem 8 MB faults 0 This actually runs slower, I believe, because it disables the line oriented optimizations that ripgrep uses. In this case, it isn't as good at detecting the `https:` literal and looking for that first. That's where `ired` can do (a lot) better, because it isn't line oriented and doesn't need to support arbitrary regex patterns. greps are. To complete this analysis, I'm going to do something that I realize is blasphemous to you and increase the input size by ten-fold. This will help us understand where time is being spent: $ time grep --color=never -Eo .{100}https:.{50} test.10x.json real 17.931 user 17.906 sys 0.017 maxmem 7 MB faults 0 $ time rg-14.0.3 --color never -N -o '.{100}https:.{50}' test.10x.json real 0.032 user 0.017 sys 0.010 maxmem 23 MB faults 0 $ time rg-14.0.3 --color always -N -o '.{100}https:.{50}' test.10x.json real 0.137 user 0.034 sys 0.019 maxmem 23 MB faults 0 $ time dash -c "echo https:|./1.sh test.10x.json 156 100" real 0.067 user 0.089 sys 0.069 maxmem 7 MB faults 0 I compared the profiles of `rg --color=never` and `rg --color=always`, and they look about the same to me. This suggests to me that color is slower simply because rendering it in my terminal emulator is slower. For grins, I also tried ugrep: $ time ugrep-4.4.1 --color=never -o '.{100}https:.{50}' test.10x.json real 6.003 user 5.977 sys 0.007 maxmem 6 MB faults 0 Owch. But not as bad as GNU grep. So with a bigger input, we can see that `rg -o` is about twice as fast as ired, even on "ordinary" hardware. And IMO, for inputs of the size you've provided, the difference is not meaningful. Going back to your original prompt: > Personally I'm interested in "grep -o" alternatives. It seems to me like `rg -o` is quite serviceable in that regard, and at the very least, substantially better than GNU grep. At this point, I wondered what ired did for substring search[2]. That immediately stuck out to me as something that looked wrong. Indeed: $ cat haystack ABAABAB $ echo -n BAB | od -An -tx1 | sed 's>^>/>;s/ //g' | ired -n haystack 0x4 $ echo -n ABAB | od -An -tx1 | sed 's>^>/>;s/ //g' | ired -n haystack $ rg -o ABAB haystack 1:ABAB So ired is a toy. One wonders how many search results you've missed over the years because of ired's feature "it's so minimal that it's wrong!" I mean sometimes tools have bugs. ripgrep has had bugs too. But this one has been in ired since 2009. What is it that you said? YIKES. Yeah. Seems appropriate. [1]: https://github.com/BurntSushi/dotfiles/blob/eace294fd80bfde13d9443c483d223d6a56cc3a2/.zshrc#L143 https://github.com/BurntSushi/dotfiles/blob/eace294fd80bfde1... [2]: https://github.com/radare/ired/blob/a1fa7904e6ad239dde950de5dc49a51be47e0c05/cmd.c#L111-L119 https://github.com/radare/ired/blob/a1fa7904e6ad239dde950de5...