3 ms·
curl -A "" https://raw.githubusercontent.com/json-iterator/test-data/0bce379832b475a6c21726ce37f971f8d849513b/large-file.json \ |tr -d '\12' > test.json Ho
by 1vuio0pswjnm7 3y ago
curl -A "" https://raw.githubusercontent.com/json-iterator/test-data/0bce379832b475a6c21726ce37f971f8d849513b/large-file.json \
|tr -d '\12' > test.json
How slow is grep -o for printing a match with surrounding characters
To observe, keep adding characters before the match
_test1(){
echo .{$1}'(https:)|(http:)'.{$2};
n=0;while test $n -le 3;do
busybox time grep -Eo ".{$1}(https:)|(http:).{$2}" test.json |sed d;
echo;
n=$((n+1));
done;
}
_test1 5 10;
_test1 15 10;
_test1 25 10;
_test1 35 10;
_test1 45 10;
_test1 55 10;
_test1 105 10;
Question: Does piping rg output to wc -l affect time(1) output?
Answer: [ ] Yes [ ] No
_test2(){
echo .{$1}'(https:)|(http:)'.{$2};
n=0;while test $n -le 3;do
busybox time rg -o .{$1}'(https:)|(http:)'.{$2} test.json;
sleep 2;
echo;
echo Now try this with a pipe to wc -l...;
echo;
sleep 2
busybox time rg -o .{$1}'(https:)|(http:)'.{$2} test.json |wc -l >/dev/null;
echo;
sleep 5;
echo;
n=$((n+1));
done;
}
_test2 150 10;
- burntsushi 3y ago> Does piping rg output to wc -l affect time(1) output? Oh yes absolutely! If `rg` is printing to a tty, it will automatically enable showing line numbers and printing with colors. Both of those have costs (over and beyond just printing matches and their byte offsets) that appear irrelevant to your use case. Neither of those things are done by ired. It's not about `wc -l` specifically, but about piping into anything. And of course, with `wc -l`, you avoid the time needed to actually render the results. But I used `wc -l` with ired too, so I "normalized" the benchmarking model and simplified it. But either way, my most recent comment before this one capitulated to your demands and avoided the use of piping results into anything. It was for this reason that I showed commands with `--color=never -N`. And yes, `grep -Eo` gets slower with more `.`. ripgrep does too, but is a bit more robust than GNU grep. I already demonstrated this in my most recent previous comment and even wrote some words about it explicitly and that regex engines can't typically do as well with increasing window sizes like this when compared to a purpose built tool like ired. Nevertheless, ired is still slower than ripgrep in most of the tests I showed in my previous comment. But optimally speaking, could something even faster than both ired and ripgrep be built? I believe so, yes. But only for some workloads, I suspect, with high match frequency. And ain't nobody going to build such a thing just to save a few milliseconds. Lol. The key is really to implement the windowing explicitly instead of relying on the regex engine to do it for you. Alternatively, one could add a special optimization pass to the regex engine that recognizes "windowing" patterns and does something clever. I have a ticket open for something similar[1]. [1]: https://github.com/rust-lang/regex/issues/802 https://github.com/rust-lang/regex/issues/802
- burntsushi 3y agoI tried your `_test2` and `rg` without `wc -l` takes 0.02s while with `wc -l` it takes 0.01s. The difference is meaningless. I don't believe you if you say that impacts your edit-compile-run cycle.
- 1vuio0pswjnm7 3y agoBelow are the results I got from _test2. + echo .{150}(https:)|(http:).{10} + n=0 + test 0 -le 3 + busybox time rg -o .{150}(https:)|(http:).{10} test.json real 1m 33.37s user 0m 1.25s sys 0m 2.97s + sleep 2 + echo + echo Now try this with a pipe to wc -l... + echo + sleep 2 + busybox time+ rg -o .{150}(https:)|(http:).{10} test.json wc -l real 0m 0.49s user 0m 0.45s sys 0m 0.02s + echo + sleep 5 + echo + n=1 + test 1 -le 3 + busybox time rg -o .{150}(https:)|(http:).{10} test.json real 1m 34.23s user 0m 1.75s sys 0m 4.22s + sleep 2 + echo + echo Now try this with a pipe to wc -l... + echo + sleep 2 + busybox time rg -o .{150}(https:)|(http:).{10} test.json wc -l real 0m 0.40s user 0m 0.37s sys 0m 0.02s + echo + sleep 5 + echo + n=2 + test 2 -le 3 + busybox time rg -o .{150}(https:)|(http:).{10} test.json real 1m 33.59s user 0m 1.05s sys 0m 1.76s + sleep 2 + echo + echo Now try this with a pipe to wc -l... + echo + sleep 2 + busybox time rg -o .{150}(https:)|(http:).{10}+ test.json wc -l real 0m 0.45s user 0m 0.37s sys 0m 0.04s + echo + sleep 5 + echo + n=3 + test 3 -le 3 + busybox time rg -o .{150}(https:)|(http:).{10} test.json real 1m 33.99s user 0m 1.93s sys 0m 4.82s + sleep 2 + echo + echo Now try this with a pipe to wc -l... + echo + sleep 2 + busybox time rg -o .{150}(https:)|(http:).{10} test.json wc -l real 0m 0.40s user 0m 0.37s sys 0m 0.02s + echo + sleep 5 + echo + n=4 + test 4 -le 3 + exit No, I am not going to stare at the screen for a minute and half as thousands of matches are displayed. (In fact I am unlikely to even be examining a file of this size. It's more likely to be under 6M.) With a file this size what I would do is examine a sample of the matches, let's say for example the first 20. Look at the speed of a shell script with 9 pipes, using ired 3x to examine the first 20 matches. busybox time sh -c "echo -n https: |od -tx1 -An \ |tr -d '\40' \ |sed 's>^>/>' \ |ired -n test.json \ |sed '1,21s/.*/s&@s-150@b166@X/;21q' \ |tr @ '\12' \ |ired -n test.json \ |sed 's/.*/w&0a/' \ |ired -n /dev/stdout" real 0m 0.01s user 0m 0.00s sys 0m 0.00s That speed is just right. Now with the same ripgrep. busybox time rg -o .{150}https:.{10} test.json|sed 20q real 0m 0.40s user 0m 0.37s sys 0m 0.02s Being used to the speed of ired, this is slow. Even more, ired is fraction of the size of ripgrep. For me, the choice of what to use is easy, until I discover something better than ired.