4 ms·
Processing UTF8 is very barely more complex, and mostly for whoever writes your language's String implementation. Once you have to worry about whether a code po
by johncolanduoni 9y ago
Processing UTF8 is very barely more complex, and mostly for whoever writes your language's String implementation. Once you have to worry about whether a code point is more than one unit, there's not much difference between 1-2 units and 1-4 units. Ideally you should be working with grapheme clusters anyway since that's the only way to have a hope of not butchering things (even non-normalized Latin text may contain multi-codepoint letters), but most languages don't give you a good way to deal with them so that's difficult in practice.
With UTF-8 you'll at least have a shot at noticing that you're not handling multi-unit codepoints well, while with UTF-16 you won't notice unless you test Chinese or a more off the beaten path language.
- mikeash 9y agoI think that last part is important, and it's a big reason why I dislike UTF-16. It's much harder to notice an inadequate implementation when you're using UTF-16. Note that most of Chinese is still in the BMP so even then it will probably work fine. You'll get failures on more obscure Chinese characters, most emoji, and really obscure scripts like linear B and cuneiform.
- millstone 9y agoUTF-8 makes it more obvious that you're mishandling multi-unit code points, BUT it introduces its own issues, specifically invalid code units and non-shortest forms. These issues represent security vulnerabilities which have been successfully exploited, and are impossible-by-design with UTF-16.