3 ms·
I completely agree with the author on this. The old Unicode 9.0 best practice made sense for a UTF-8 decoder that consumed input using a state machine, and like
by rossy 9y ago
I completely agree with the author on this. The old Unicode 9.0 best practice made sense for a UTF-8 decoder that consumed input using a state machine, and like the article says, if you accept the correct byte-ranges in each state as in Table 3-7, your state-machine-based decoder will implicitly reject every kind of invalid sequence, including overlong encodings, encoded UTF-16 surrogates and encoded out of range characters. A state machine also makes other things trivial, like validating UTF-8 input without outputting characters and writing a streaming UTF-8 to UTF-16 converter. Another property is that it rejects invalid input as soon as possible, eg. as soon as it consumes input that can't possibly be part of an invalid sequence.
This is an example of a particularly elegant UTF-8 decoder which uses a state machine:
http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ http://bjoern.hoehrmann.de/utf-8/decoder/dfa/
The proposed Unicode 11 best practice suits a decoder that checks for overlong sequences, encoded surrogates and out of range values as a sort of post-validation step, after consuming all the bytes of a potentially valid UTF-8 sequence. Not only is this different to the behaviour of every existing decoder except for ICU, it also seems less elegant, more complicated and more error prone to me. If I understand correctly, even bytes like 0xc0 and 0xf5, which never form part of a valid UTF-8 sequence, won't be rejected immediately in this kind of decoder.
The article makes a pretty solid argument for why this difference in behaviour matters, even though it's just a best practice and not an official requirement. The two key points in this for me are that most existing UTF-8 decoders produce identical results matching the Unicode 9.0 best practice, and that there was an actual bug in Chrome when two internal UTF-8 decoders produced differing results. I think I'd make a stronger conclusion though: Not only should they keep the current recommended best practice, they should also elevate it to a requirement in order to prevent bugs like the one in Chrome from happening in future. Most existing UTF-8 decoders are already compliant, and it would be a nice property of UTF-8 if all byte sequences, including invalid ones, decoded to the same sequence of codepoints in every decoder.