4 ms·
This is plain decoding of a trusted string: utf16 decode if (unit <= 0xD7FF || unit >= 0xE000) /* one unit */; else /* two unit
by Mikhail_Edoshin 4y ago
This is plain decoding of a trusted string:
utf16 decode
if (unit <= 0xD7FF || unit >= 0xE000)
/* one unit */;
else
/* two units */
utf-8 decode
if (unit < 0xE0) {
if (unit < 0xC0)
/* one unit */
else
/* two units */
}
else if (unit < 0xF0)
/* three units */
else
/* four units */
UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding:
utf16 encode
if (value <= 0xFFFF)
/* one unit */
else
/* two units */
utf8 encode
if (value <= 0x7FF) {
if (value <= 0x7F)
/* one unit */
else
/* two units */
}
else if (value <= 0xFFFF)
/* three units */
else
/* four units */
(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)
Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:
UTF-8 : 13 byte types, 13 states, 10 actions
UTF-16LE : 3 byte types, 4 states, 5 actions
- SaddledBounding 4y agoutf-8 has an elegance that utf-16 lacks: utf-8 decode switch (std::countl_one(unit)) { case 0: /* one unit */ break; case 2: /* two units */ break; case 3: /* three units */ break; case 4: /* four units */ break; default: /* not code point boundary */ break; }
- Mikhail_Edoshin 4y agoOh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
- bergesenha 4y agoI’m happy every time I see c++20 in the wild