2 ms·
Beyond the issue noted with UTF-16 permitting surrogate pairs, it's also important to understand that UTF-16 does not free you from normalization form concerns.
by mpyne 12y ago
Beyond the issue noted with UTF-16 permitting surrogate pairs, it's also important to understand that UTF-16 does not free you from normalization form concerns.
For instance, even within the 16-bit Basic Multilingual Plane, it is not safe to reverse a UTF-16 string by simply reversing each block of 2 bytes, as a Unicode glyph (e.g. á) can be composed of a pre-combined code point which only takes 16-bits, or a base character (a) followed by a combiner (´).
The pre-combined variant would reverse just fine, but the renderer-combined á would reverse to ´a, which is probably not what was intended.
If you're dealing with Unicode you need to be dealing with a good library and you need to be understanding what you're doing, once you go away from simply reading/displaying/serializing bags-o-bits.
But if the latter is all you're doing, UTF-8 is just fine, and less susceptible to mistakes with code that works correctly on C strings.