4 ms·
UTF-16 is a variable-width encoding. Thanks to surrogate pairs some code points take 2 bytes, and some take 4. Even if you use UCS-2 (the 2-byte "UTF-16" infamo
by pornel 9y ago
UTF-16 is a variable-width encoding. Thanks to surrogate pairs some code points take 2 bytes, and some take 4. Even if you use UCS-2 (the 2-byte "UTF-16" infamous for mangling emoji) constant-time indexing of code points still doesn't give you constant time access to what humans would call characters ("grapheme clusters" in Unicode), because of decomposed characters with modifiers and joiners.
Languages that use UTF-16 mostly use it only because they're older than UTF-8 (or are tied to a platform older than UTF-8).
- millstone 9y agoUTF-16 is a variable width encoding that is also more convenient to process compared to UTF-8. For example you don't have to deal with invalid code units, non-shortest forms, etc. I think you're right that older languages use UTF-16 and newer ones use UTF-8. But it also seems empirically true that UTF-16 languages do better at grappling with Unicode's subtleties, compared to UTF-8. The temptation of UTF8-is-bsically-C-strings is hard to ignore.
- duskwuff 9y ago> For example you don't have to deal with invalid code units, non-shortest forms, etc. You have to deal with unpaired surrogates, though. And just because that wasn't annoying enough, any UTF-8 sequence which contains an encoded surrogate (which is technically invalid, but not prohibited by most implementations) is impossible to encode as UTF-16.