4 ms·

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingu

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.

Const-me 4y ago
UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.
- barrkel 4y ago
  That's 33% less memory, or conversely, UTF-8 uses 50% more memory.
Mikhail_Edoshin 4y ago
This is plain decoding of a trusted string: utf16 decode if (unit <= 0xD7FF || unit >= 0xE000) /* one unit */; else /* two units */ utf-8 decode if (unit < 0xE0) { if (unit < 0xC0) /* one unit */ else /* two units */ } else if (unit < 0xF0) /* three units */ else /* four units */ UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode if (value <= 0xFFFF) /* one unit */ else /* two units */ utf8 encode if (value <= 0x7FF) { if (value <= 0x7F) /* one unit */ else /* two units */ } else if (value <= 0xFFFF) /* three units */ else /* four units */ (We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.) Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats: UTF-8 : 13 byte types, 13 states, 10 actions UTF-16LE : 3 byte types, 4 states, 5 actions
- SaddledBounding 4y ago
  utf-8 has an elegance that utf-16 lacks: utf-8 decode switch (std::countl_one(unit)) { case 0: /* one unit */ break; case 2: /* two units */ break; case 3: /* three units */ break; case 4: /* four units */ break; default: /* not code point boundary */ break; }
  - Mikhail_Edoshin 4y ago
    Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
  - bergesenha 4y ago
    I’m happy every time I see c++20 in the wild

Const-me 4y ago

UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.

barrkel 4y ago
That's 33% less memory, or conversely, UTF-8 uses 50% more memory.

Mikhail_Edoshin 4y ago

This is plain decoding of a trusted string: utf16 decode if (unit <= 0xD7FF || unit >= 0xE000) /* one unit */; else /* two units */ utf-8 decode if (unit < 0xE0) { if (unit < 0xC0) /* one unit */ else /* two units */ } else if (unit < 0xF0) /* three units */ else /* four units */ UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode if (value <= 0xFFFF) /* one unit */ else /* two units */ utf8 encode if (value <= 0x7FF) { if (value <= 0x7F) /* one unit */ else /* two units */ } else if (value <= 0xFFFF) /* three units */ else /* four units */ (We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.) Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats: UTF-8 : 13 byte types, 13 states, 10 actions UTF-16LE : 3 byte types, 4 states, 5 actions

SaddledBounding 4y ago
utf-8 has an elegance that utf-16 lacks: utf-8 decode switch (std::countl_one(unit)) { case 0: /* one unit */ break; case 2: /* two units */ break; case 3: /* three units */ break; case 4: /* four units */ break; default: /* not code point boundary */ break; }
- Mikhail_Edoshin 4y ago
  Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
- bergesenha 4y ago
  I’m happy every time I see c++20 in the wild