3 ms·

> First he doesn't know about ripgrep ripgrep isn't POSIX compliant. Never was and never will be. So I don't think it's really applicable here. ripgrep is mayb

> First he doesn't know about ripgrep ripgrep isn't POSIX compliant. Never was and never will be. So I don't think it's really applicable here. ripgrep is maybe an existence proof that a competing tool can be written, but it is certainly not a suitable POSIX compliant grep replacement. Building a fully POSIX compliant grep tool with good performance like GNU grep is pretty difficult. It could be done. It would probably take me a couple months (and only by leveraging my existing work). I just don't have any incentive to do it, personally. > Second,none of the known grep's can find unicode strings. That's definitely not true. GNU grep can do this just fine: $ echo 'Δ' | LC_ALL=en_US.UTF-8 grep '\w' Δ $ echo 'Δ' | LC_ALL=en_US.UTF-8 grep 'δ' -i Δ GNU grep does pay more of a performance penalty for Unicode support than ripgrep does. It's easy to see this with a somewhat pathological case by comparing `rg '\w{42}'` with `LC_ALL=en_US.UTF-8 grep -E '\w{42}'`. (Not all cases are pathological, but ripgrep and GNU grep are both very good about literal optimizations, so it's easiest to demonstrate with a pathological case.) I'm not sure if one can attribute this to better library support in the Rust ecosystem though. It's really about how the regex machinery itself is built.

rurban 7y ago
Unicode search is not searching for byte equivalents. Unicode characters can be composed in different ways, with different bytes. What you see is a á but internally it can be composed of different bytes. the single a with accent or with a mark, a plus accent. Unicode search needs do be optimized to normalize characters, and similar problems exist for case folding, grep -i These problems are also security relevant btw. Many unicode strings are now identifiers, like names or file paths. that the stdlib still provides no functions to search and compare strings is a much bigger beef. grep is just a symptom of a much bigger problem
- burntsushi 7y ago
  Yes, you mentioned normalization and I didn't correct that, because indeed, grep can't do that. But you also just said "Unicode search," which is a pretty general term. GNU grep is certainly Unicode aware, as I demonstrated. It just doesn't solve every use case you want. But it does have Unicode-aware character classes and also Unicode aware case folding. A grep tool that did normalization would be extremely slow. You'd probably be better served by a specialized tool. Or even better, it should be possible to write a fairly simple wrapper that puts everything into your desired normal form before searching.

rurban 7y ago

Unicode search is not searching for byte equivalents. Unicode characters can be composed in different ways, with different bytes. What you see is a á but internally it can be composed of different bytes. the single a with accent or with a mark, a plus accent. Unicode search needs do be optimized to normalize characters, and similar problems exist for case folding, grep -i These problems are also security relevant btw. Many unicode strings are now identifiers, like names or file paths. that the stdlib still provides no functions to search and compare strings is a much bigger beef. grep is just a symptom of a much bigger problem

burntsushi 7y ago
Yes, you mentioned normalization and I didn't correct that, because indeed, grep can't do that. But you also just said "Unicode search," which is a pretty general term. GNU grep is certainly Unicode aware, as I demonstrated. It just doesn't solve every use case you want. But it does have Unicode-aware character classes and also Unicode aware case folding. A grep tool that did normalization would be extremely slow. You'd probably be better served by a specialized tool. Or even better, it should be possible to write a fairly simple wrapper that puts everything into your desired normal form before searching.