5 ms·
OK, so I'll try your commands: $ busybox time grep -Eo .{35}https:.{4} test.json real 0m 0.15s user 0m 0.15s sys 0m 0.00s $ busy
by burntsushi 3y ago
OK, so I'll try your commands:
$ busybox time grep -Eo .{35}https:.{4} test.json
real 0m 0.15s
user 0m 0.15s
sys 0m 0.00s
$ busybox time rg-14.0.3 -o .{35}https:.{4} test.json
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
$ busybox time dash -c "echo https:|./1.sh test.json 45 35"
real 0m 0.01s
user 0m 0.01s
sys 0m 0.00s
$ busybox time bash -c "echo https:|./1.sh test.json 45 35"
real 0m 0.00s
user 0m 0.00s
sys 0m 0.00s
$ busybox time dash -c "echo https:|./busy-1.sh test.json 45 35"
real 0m 0.00s
user 0m 0.01s
sys 0m 0.00s
$ busybox time bash -c "echo https:|./busy-1.sh test.json 45 35"
real 0m 0.01s
user 0m 0.01s
sys 0m 0.00s
So grep -o takes 150ms, but both ripgrep and ired are seemingly instant. But if I use zsh's builtin `time` command with my own TIMEFMT[1], it gives me numbers greater than 0:
$ time grep -Eo .{35}https:.{4} test.json
real 0.324
user 0.317
sys 0.007
maxmem 16 MB
faults 0
$ time rg-14.0.3 -o .{35}https:.{4} test.json
real 0.008
user 0.003
sys 0.003
maxmem 16 MB
faults 0
$ time dash -c "echo https:|./1.sh test.json 45 35"
real 0.010
user 0.011
sys 0.007
maxmem 16 MB
faults 0
$ time bash -c "echo https:|./1.sh test.json 45 35"
real 0.011
user 0.014
sys 0.004
maxmem 16 MB
faults 0
Would you look at that. ripgrep is faster! By a whole 2 milliseconds! WOW!
OK, since I'm a software developer and thus apparently cannot understand the lowly needs of an "ordinary user," I'll hop over to my machine with a i5-7600, which was released 6 years ago. Is that ordinary enough, or still too super charged to do any meaningful comparison whatsoever?
$ time grep -Eo .{35}https:.{4} test.json
real 0.641
user 0.620
sys 0.017
maxmem 6 MB
faults 0
$ time rg-14.0.3 -o .{35}https:.{4} test.json
real 0.010
user 0.008
sys 0.000
maxmem 8 MB
faults 0
$ time dash -c "echo https:|./1.sh test.json 45 35"
real 0.011
user 0.009
sys 0.011
maxmem 6 MB
faults 0
$ time bash -c "echo https:|./1.sh test.json 45 35"
real 0.013
user 0.021
sys 0.003
maxmem 6 MB
faults 0
(I ran the commands above each several times and took the minimum.)
OK, so ripgrep is still 1ms faster even on "ordinary user" hardware.
All right, so your other comment also shared another benchmark:
$ time grep -Eo .{100}https:.{50} test.json
real 1.777
user 1.772
sys 0.003
maxmem 6 MB
faults 0
$ time rg-14.0.3 -o .{100}https:.{50} test.json
real 0.013
user 0.006
sys 0.000
maxmem 8 MB
faults 0
$ time rg-14.0.3 --color never -o .{100}https:.{50} test.json
real 0.006
user 0.006
sys 0.000
maxmem 8 MB
faults 0
$ time dash -c "echo https:|./1.sh test.json 156 100"
real 0.015
user 0.024
sys 0.004
maxmem 7 MB
faults 0
$ time bash -c "echo https:|./1.sh test.json 156 100"
real 0.016
user 0.028
sys 0.000
maxmem 7 MB
faults 0
(Notice that disabling color and line numbers for ripgrep improves its speed a fair bit. ired isn't doing either of those things, so it's only fair. GNU grep doesn't count line numbers by default and disabling color doesn't improve its perf here.)
This one is more interesting because it exposes the fact that many regex engines have trouble dealing with bounded repeats. Something like `.{100}` for example is not executed particularly efficiently in most regex engines. And indeed, in ripgrep by default, `.` actually matches the UTF-8 encoding of any Unicode scalar value (so between 1 and 4 bytes) and not any arbitrary byte. You'd need to pass the `--no-unicode` flag or prefix your pattern with `(?-u)` to match any arbitrary byte. And indeed, even then, `.` doesn't match `\n`. So you might even want `(?s-u)`. But since this is a grep and *greps are line oriented*, you'd need to enable multi-line mode in ripgrep (GNU grep doesn't have this):
$ time rg-14.0.3 -Uo '(?s-u).{100}https:.{50}' test.json
real 0.057
user 0.041
sys 0.006
maxmem 8 MB
faults 0
$ time rg-14.0.3 --color never -N -Uo '(?s-u).{100}https:.{50}' test.json
real 0.042
user 0.041
sys 0.000
maxmem 8 MB
faults 0
This actually runs slower, I believe, because it disables the line oriented optimizations that ripgrep uses. In this case, it isn't as good at detecting the `https:` literal and looking for that first. That's where `ired` can do (a lot) better, because it isn't line oriented and doesn't need to support arbitrary regex patterns. greps are.
To complete this analysis, I'm going to do something that I realize is blasphemous to you and increase the input size by ten-fold. This will help us understand where time is being spent:
$ time grep --color=never -Eo .{100}https:.{50} test.10x.json
real 17.931
user 17.906
sys 0.017
maxmem 7 MB
faults 0
$ time rg-14.0.3 --color never -N -o '.{100}https:.{50}' test.10x.json
real 0.032
user 0.017
sys 0.010
maxmem 23 MB
faults 0
$ time rg-14.0.3 --color always -N -o '.{100}https:.{50}' test.10x.json
real 0.137
user 0.034
sys 0.019
maxmem 23 MB
faults 0
$ time dash -c "echo https:|./1.sh test.10x.json 156 100"
real 0.067
user 0.089
sys 0.069
maxmem 7 MB
faults 0
I compared the profiles of `rg --color=never` and `rg --color=always`, and they look about the same to me. This suggests to me that color is slower simply because rendering it in my terminal emulator is slower.
For grins, I also tried ugrep:
$ time ugrep-4.4.1 --color=never -o '.{100}https:.{50}' test.10x.json
real 6.003
user 5.977
sys 0.007
maxmem 6 MB
faults 0
Owch. But not as bad as GNU grep.
So with a bigger input, we can see that `rg -o` is about twice as fast as ired, even on "ordinary" hardware.
And IMO, for inputs of the size you've provided, the difference is not meaningful.
Going back to your original prompt:
> Personally I'm interested in "grep -o" alternatives.
It seems to me like `rg -o` is quite serviceable in that regard, and at the very least, substantially better than GNU grep.
At this point, I wondered what ired did for substring search[2]. That immediately stuck out to me as something that looked wrong. Indeed:
$ cat haystack
ABAABAB
$ echo -n BAB | od -An -tx1 | sed 's>^>/>;s/ //g' | ired -n haystack
0x4
$ echo -n ABAB | od -An -tx1 | sed 's>^>/>;s/ //g' | ired -n haystack
$ rg -o ABAB haystack
1:ABAB
So ired is a toy. One wonders how many search results you've missed over the years because of ired's feature "it's so minimal that it's wrong!" I mean sometimes tools have bugs. ripgrep has had bugs too. But this one has been in ired since 2009.
What is it that you said? YIKES. Yeah. Seems appropriate.
[1]: https://github.com/BurntSushi/dotfiles/blob/eace294fd80bfde13d9443c483d223d6a56cc3a2/.zshrc#L143 https://github.com/BurntSushi/dotfiles/blob/eace294fd80bfde1...
[2]: https://github.com/radare/ired/blob/a1fa7904e6ad239dde950de5dc49a51be47e0c05/cmd.c#L111-L119 https://github.com/radare/ired/blob/a1fa7904e6ad239dde950de5...