5 ms·
Wait. Does it actually validate utf8 or just checks ranges in which 1-4 byte characters are encoded?
by karlisolte 8y ago
Wait. Does it actually validate utf8 or just checks ranges in which 1-4 byte characters are encoded?
- kevin_thibedeau 8y agoBjörn Höhrmann wrote the gold standard for a minimalist scalar decoder and didn't depend on UB to do it: https://bjoern.hoehrmann.de/utf-8/decoder/dfa/ https://bjoern.hoehrmann.de/utf-8/decoder/dfa/
- petermcneeley 8y agoBjörn's decoder is nearly identical to the naive validity checker found here. It is about 9x slower than the superposition lookup table found here.
- jepler 8y agoI gave it a whirl and it correctly rejects examples of each 'invalid byte sequence' cited by wikipedia. "\xff", // invalid byte "\x82", // unexpected continuation byte "\xc0", // not enough continuation bytes "\xed\xa0", // not enough continuation bytes "\xc0\x80", // "modified UTF-8's not really NUL (over-long encoding) "\xf0\x82\x82\x8c", // over-long euro symbol "\xed\xa0\x80", // invalid surrogate character U+d800
- kwillets 8y agoMost of the checks reduce to range checks, eg overlong sequences have 0's in the low-order bits of the first byte, which means 0xE0 is invalid. I don't think he missed anything.