4 ms·
task: printing non-repeating patterns in relatively small files to the screen, optionally with some context context should be printed exactly as it appears in
by 1vuio0pswjnm7 3y ago
task: printing non-repeating patterns in relatively small files to the screen, optionally with some context
context should be printed exactly as it appears in the file, i.e., newlines should be printed
ired vs ripgrep, which one is better suited for this task
one uses regular expressions, the other does not
one is a 76k static binary that fits in 2MB L2 cache, the other is a 5.7MB dynamically-linked binary
2 shell scripts to demonstrate differences
usage: echo pattern|1.sh [num chars before] [num chars after]
1. "1.sh" using 5.7MB binary, PCRE2
#!/bin/sh
read x;
case $# in :)
;;0)exec echo "usage: ${0##*/} file [num chars before] [num chars after]"
;;1)exec rg -uuu -no-unicode --block-buffered --color=never -NUo "$x" $1
esac
case $# in 2|3)printf "((.)|(\\\\n)|(\\\\r)){"$2"}$x((.)|(\\\\n)|(\\\\r)){"${3-0}"}";esac \
|rg -f/dev/stdin -uuu --no-unicode --block-buffered --color=never -NUo $1
2. "2.sh" using 76k static binary, no regular expressions
#!/bin/sh
read x;
len=${#x};
b=$(($2+$len));
case $# in 0)exec echo "usage ${0##*/} file [num chars before] [num chars after]"
;;2)b=$(($2+$len))
;;3)b=$(($3+$2+$len))
esac
echo "$x" > .x
{ printf /;ired -n -c X1 .x;} \
|ired -n $1 \
|sed "s/.*/s&@s-${2-0}@b$b@X/;" \
|tr @ '\12' \
|ired -n $1 \
|sed 's/.*/w&0a/' \
|ired -n /dev/stdout \
|sed -e '/^Invalid hexpair/d'
generate test data:
curl -4si0 -A "" https://www.google.com > test.html
usage example: find the pattern "(" in test.html, display results to screen
echo \(|1.sh test.html
regex parse error:
(?:()
^
error: unclosed group
echo '[(]'|1.sh test.html
echo \(|2.sh test.html
cat .x
observation: the task is simple but 1.sh may require more typing and knowledge of regular expressions
observation: 2.sh does not require knowledge of PCRE; the pattern requires no extra chars, e.g., brackets
usage example: find the pattern "(a" in test.html, display results to screen with 0 chars before and 3 chars after
echo '[(]a'|1.sh test.html 0 3|sed -n l|less -N
echo \(a|2.sh test.html 0 3|sed -n l|less -N
observation: 1.sh does not include the newline after match #187; some workaround is required for 1.sh
conclusion: for me, ripgrep is too large and complicated for this simple task involving relatively small files; it's overkill. it does not feel any faster than ired at the command line. in fact, it feels slower. like python or java, or other large rust/go binaries, there is a small initial delay, a jank. whereas ired feels very smooth.
- 1vuio0pswjnm7 3y agoThe failure of grep/ripgrep to display the newline character contained in the context in match #178 could be characterised as a "false negative".
- burntsushi 3y agoWrong again. The . doesn't match newlines by default. That's standard for regex. As with just about any regex engine, you need to explicitly instruct . to match newlines. In ripgrep, that means `(?s)..query..` for example. Or pass `--multiline-dotall`. Or use `\p{any}` to match the UTF-8 encoding of any Unicode scalar value. Or `(?-u:[\x00-\xFF])` to match any byte value. In contrast, not being able to find ABAB in ABAABAB is definitely a false negative. There's no qualification needed there. You're using badly broken software. One wonders why you can't seem to bring yourself to acknowledge it as a downside.
- 1vuio0pswjnm7 3y agoYou have failed to provide a working example that does the simple task I presented, which ired does perfectly. That's all I am looking for. A working example. You also claimed you would provide an explanation of why traditional grep -o is so slow when adding preceeding context, e.g., .{500}pattern. You never did. You decided to start whining incessantly about not being able to match substrings instead. Again, I am not searching for repeating strings such as ABAB. I am searching for AB preceded or followed by unknown characters. Thus the "problem" you found is not one I am going to have given the way I'm using the program. Why would I care about it. It's a hex editor, not a program I use to do searches. I never searched for a repeating pattern. You did. Then you proceed to whine about it. Incessantly. Hoping to create some sort of diversion. Further, if you were really paying attention, you would notice that in the scripts I presented I'm not using ired to search for strings (/"string"). I am searching multiple, catenated hex pairs (/737472696e67). ired is not intended to be used that way. Although it works for my purposes, ired is only intended to be used to search a single hex pair. The issue you spotted is applicable when searching _strings_, not searching a single hex pair. But I am not searching strings. I'm searching multiple hex pairs. And because the program is not intended to be used to match catenated hex pairs, I fully expected this might not work. As it happens, it works. Unless an example is provided, it appears that using rg -o to _print_ (not just match) characters found trailing context that happen to be newlines (task #1) works about as well as using ired to search for repeating strings in a file (task #2). It does not work. This is not surprising to me since IMO there are better programs to do those tasks. As stated in the very first comment I made, I am interesed in programs that perform task #1. I do not need solutions for task #2. Despite what you may be selling. You cannot produce a working example.