3 ms·
Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs ac
by barrkel 15y ago
Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs accented characters. To do the Right Thing (tm), you need to normalize, and that adds at least another pass over the whole string.
If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.
- _delirium 15y agoIn addition, conforming Unicode implementations have to ensure that their input is well-formed Unicode, and reject non-conforming characters (for some security-related reasons). So the original goal of UTF8, that it could be usable for straight byte-level search (the UTF8 RFC mentions a goal of allowing Boyer-Moore to be applied directly) is no longer really possible. edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932 http://news.ycombinator.com/item?id=2860932
- Confusion 15y agoIt's not necessarily grep's responsibility to normalize strings. They could very well stipulate that it is the responsibility of the user to make sure the both the input and the search expression are already normalized. That way, the normalization could be done by a separate program, resulting in the reusability and separation of concerns that is The Unix Way.
- barrkel 15y agoI fully agree, if you redefine grep as a byte search program, rather than for text search. Rule it out for most languages apart from English. To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched: #!/usr/bin/bash regex="$1"; shift unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")"