3 ms·
The article is correct if by substring you mean exact sequence of code points. This is still remarkable, because even that is not always possible in variable-wi
by pornel 16y ago
The article is correct if by substring you mean exact sequence of code points. This is still remarkable, because even that is not always possible in variable-width encodings.
Problem of searching for semantically equivalent text (taking into account decomposed forms, ligatures, etc.) is higher-level and applies to all Unicode encodings (i.e. using UCS-2 or UTF-32 doesn't solve it either).
- chronomex 16y agoDanger, Danger Will Robinson! UTF-8 strings can include overlong encodings (e.g. using 3 bytes to encode a code point that's normally encoded in 1 byte). The Unicode standard disallows creation of these forms, but until recently (http://www.unicode.org/versions/corrigendum1.html http://www.unicode.org/versions/corrigendum1.html), only "strongly discouraged" interpretation of them. I do not doubt that there is code in the wild which will interpret them as characters.
- justin_vanw 16y agoRecommending that people manipulate utf-8 directly is a bad idea for lots of reasons. It's fairly easy to end up with mixed normalizations and lots of other things which will come back to bite you. Just about everything, including GNU command line utilities, support proper unicode handling, why would you want to go back into the stone ages of dealing with bytes directly?