3 ms·
Harmful, not really. Sometimes UTF-16 is the most compact representation for a given string, and its semantics are not that confusing. UTF-8 is a better "defa
by jrockway 15y ago
Harmful, not really. Sometimes UTF-16 is the most compact representation for a given string, and its semantics are not that confusing. UTF-8 is a better "default choice", but the algorithms for both are very hard to get right. Given an idealized Unicode string object, it's still hard to do things like count the number of glyphs required to display the text in font foo, or convert all characters to "lowercase", or sort a list of names in "alphebetical" order. But the problem is not Unicode or UTF-16 or UTF-8, the problem is defining what a correct computer program is, and ultimately, that's the job for the programmer, not the string encoding algorithm.
Think of UTF-8 and UTF-16 as being like in-memory compression for Unicode strings. You can't take a gzip file, count the number of bytes, and multiply by some factor to get the true length of the string. But, this complexity is often worth the space savings, so that's why these encodings exist.
The problem that people have is twofold: one, the average programmer doesn't realize that characters and character encodings are two different things. The second is, they don't realize that a Unicode string is a data structure that does one thing and only one thing: maintain a sequence of Unicode codepoints in memory.
A Unicode string is not "text" in some language; it's some characters for a computer. To choose the right font or sort things correctly, you need additional information sent out-of-band. There is no way to do it right given only the Unicode string itself. People are reluctant to send out-of-bad data and so other people assume Unicode is hard. It's not. The problem is that Unicode is not text.