4 ms·

This article is retarded in lots of ways, but I'll quickly debunk one point. 5. Substring search is just byte string search. Nope. The same character sequenc

This article is retarded in lots of ways, but I'll quickly debunk one point. 5. Substring search is just byte string search. Nope. The same character sequences can have multiple utf-8 representations. Yes, this comes up in the real world, especially on the web. Reference: http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_form... The only way to sanely handle unicode is to load it using a codec in the language of your choice, and manipulate it using the tools your language supplies for manipulating strings OR read all the insane specs for unicode and implement your own. Treating unicode as bytes will end in tears.

pornel 16y ago
The article is correct if by substring you mean exact sequence of code points. This is still remarkable, because even that is not always possible in variable-width encodings. Problem of searching for semantically equivalent text (taking into account decomposed forms, ligatures, etc.) is higher-level and applies to all Unicode encodings (i.e. using UCS-2 or UTF-32 doesn't solve it either).
- chronomex 16y ago
  Danger, Danger Will Robinson! UTF-8 strings can include overlong encodings (e.g. using 3 bytes to encode a code point that's normally encoded in 1 byte). The Unicode standard disallows creation of these forms, but until recently (http://www.unicode.org/versions/corrigendum1.html http://www.unicode.org/versions/corrigendum1.html), only "strongly discouraged" interpretation of them. I do not doubt that there is code in the wild which will interpret them as characters.
- justin_vanw 16y ago
  Recommending that people manipulate utf-8 directly is a bad idea for lots of reasons. It's fairly easy to end up with mixed normalizations and lots of other things which will come back to bite you. Just about everything, including GNU command line utilities, support proper unicode handling, why would you want to go back into the stone ages of dealing with bytes directly?
jerf 16y ago
If you are doing any comparing of Unicode strings, or anything beyond the simplest manipulation of them, I recommend running them through the appropriate-for-your-task normalization process: http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ You should understand what the normalization forms are. Which I say despite having only a bit of a vague idea myself. But I feel bad about it. (And actually plan on fixing it soon for one of my projects anyhow.)

pornel 16y ago

The article is correct if by substring you mean exact sequence of code points. This is still remarkable, because even that is not always possible in variable-width encodings. Problem of searching for semantically equivalent text (taking into account decomposed forms, ligatures, etc.) is higher-level and applies to all Unicode encodings (i.e. using UCS-2 or UTF-32 doesn't solve it either).

chronomex 16y ago
Danger, Danger Will Robinson! UTF-8 strings can include overlong encodings (e.g. using 3 bytes to encode a code point that's normally encoded in 1 byte). The Unicode standard disallows creation of these forms, but until recently (http://www.unicode.org/versions/corrigendum1.html http://www.unicode.org/versions/corrigendum1.html), only "strongly discouraged" interpretation of them. I do not doubt that there is code in the wild which will interpret them as characters.
justin_vanw 16y ago
Recommending that people manipulate utf-8 directly is a bad idea for lots of reasons. It's fairly easy to end up with mixed normalizations and lots of other things which will come back to bite you. Just about everything, including GNU command line utilities, support proper unicode handling, why would you want to go back into the stone ages of dealing with bytes directly?

jerf 16y ago

If you are doing any comparing of Unicode strings, or anything beyond the simplest manipulation of them, I recommend running them through the appropriate-for-your-task normalization process: http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ You should understand what the normalization forms are. Which I say despite having only a bit of a vague idea myself. But I feel bad about it. (And actually plan on fixing it soon for one of my projects anyhow.)