4 ms·
I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, li
by ainar-g 4y ago
I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8.
The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.
- Mikhail_Edoshin 4y agoNo, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibility.
- heinrich5991 4y agoCan you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
- Const-me 4y agoUTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.
- barrkel 4y agoThat's 33% less memory, or conversely, UTF-8 uses 50% more memory.
- Mikhail_Edoshin 4y agoThis is plain decoding of a trusted string: utf16 decode if (unit <= 0xD7FF || unit >= 0xE000) /* one unit */; else /* two units */ utf-8 decode if (unit < 0xE0) { if (unit < 0xC0) /* one unit */ else /* two units */ } else if (unit < 0xF0) /* three units */ else /* four units */ UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode if (value <= 0xFFFF) /* one unit */ else /* two units */ utf8 encode if (value <= 0x7FF) { if (value <= 0x7F) /* one unit */ else /* two units */ } else if (value <= 0xFFFF) /* three units */ else /* four units */ (We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.) Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats: UTF-8 : 13 byte types, 13 states, 10 actions UTF-16LE : 3 byte types, 4 states, 5 actions
- SaddledBounding 4y agoutf-8 has an elegance that utf-16 lacks: utf-8 decode switch (std::countl_one(unit)) { case 0: /* one unit */ break; case 2: /* two units */ break; case 3: /* three units */ break; case 4: /* four units */ break; default: /* not code point boundary */ break; }
- Mikhail_Edoshin 4y agoOh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
- bergesenha 4y agoI’m happy every time I see c++20 in the wild
- bigDinosaur 4y agoCan you show some concrete instances of it being easier to work with compared to UTF-8? As far as I can see you gain variable length encoding (albeit in a potentially confusing way) while losing ASCII support and also not gaining the pure simplicity of UTF-32. Oh and also endianness, although personally that's probably never an issue I will run into.
- Grame1983 4y ago[dead]
- Mikhail_Edoshin 4y ago(Please see my reply to a sibling comment.)
- bigDinosaur 4y agoPerfect, thanks for the heads up!