3 ms·
Wikipedia has a summary of comparisons: https://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16 https://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16 Advantag
by mjevans 10y ago
Wikipedia has a summary of comparisons:
https://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16 https://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16
Advantages
* Byte encodings and UTF-8 are represented by byte arrays in programs, and often nothing needs to be done to a function when converting from a byte encoding to UTF-8. UTF-16 is represented by 16-bit word arrays, and converting to UTF-16 while maintaining compatibility with existing ASCII-based programs (such as was done with Windows) requires every API and data structure that takes a string to be duplicated, one version accepting byte strings and another version accepting UTF-16.
Text encoded in UTF-8 will be smaller than the same text encoded in UTF-16 if there are more code points below U+0080 than in the range U+0800..U+FFFF. This is true for all modern European languages.
Most communication and storage was designed for a stream of bytes. A UTF-16 string must use a pair of bytes for each code unit:
* * The order of those two bytes becomes an issue and must be specified in the UTF-16 protocol, such as with a byte order mark.
* * If an odd number of bytes is missing from UTF-16, the whole rest of the string will be meaningless text. Any bytes missing from UTF-8 will still allow the text to be recovered accurately starting with the next character after the missing bytes.
Disadvantages
* Characters U+0800 through U+FFFF use three bytes in UTF-8, but only two in UTF-16. As a result, text in (for example) Chinese, Japanese or Hindi will take more space in UTF-8 if there are more of these characters than there are ASCII characters. This happens for pure text[nb 2] but actual documents often contain enough spaces and line terminators, numbers (digits 0–9), and HTML or XML or wiki markup characters, that they are shorter in UTF-8. For example, both the Japanese UTF-8 and the Hindi Unicode articles on Wikipedia take more space in UTF-16 than in UTF-8.[nb 3]
- millstone 10y agoIt doesn't mention the biggest disadvantages of UTF-8 relatives to UTF-16: the existence of non-shortest forms and invalid code units.
- syncsynchalt 10y agoAh, but those are disallowed by spec and in the former case you'd open a security bug against anything that didn't transform it to U+FFFD or similar (eg. it lets you sneak '\0's into C-style strings and '/'s into unix paths). So I'll grant you a point but could match that against similar problems in UTF-16 (bad surrogate pairs, surrogate singletons, BOM bombs, and the same invalid code units).
- deathanatos 10y agoThe biggest disadvantage of UTF-16, IMO, is that programmers blindly assume that they can index into the string as if it were an array, and get a code point out — which you can not; you'll get a code unit, which is slightly different, and might not represent a full code point (let alone a full character). UTF-8's very encoding quickly beats this out of anyone who tries, whereas it's easy to eek by in UTF-16. The real problem is that the APIs allow such tomfoolery. (Some have historical excuses, I will grant, but new languages are still made that allow indexing into code units without it being obvious that this is probably not what the coder wants.)
- int_19h 10y agoStrictly speaking, this is a disadvantage of languages that have strings as first-class types and allow indexing on strings in the first place (and specify it to have this semantics). For the most part, the developer shouldn't really care about the internal encoding of the string, but the language/library should also not expose that to them.