4 ms·
Yes, the most important operation: string validation! UTF-16's validation concerns are: 1. Broken surrogate pairs, which is mostly benign. 2. Byte-order conf
by millstone 9y ago
Yes, the most important operation: string validation!
UTF-16's validation concerns are:
1. Broken surrogate pairs, which is mostly benign.
2. Byte-order confusion.
While UTF-8 has:
1. Invalid code points, for example, code points for surrogate halves.
2. Invalid code units, such as 0xFF.
3. Non-shortest forms, where a character may be encoded multiple ways.
4. Representation of NUL, and potential for confusion with APIs that expect null-terminated strings.
In practice the UTF-8 issues have caused much more serious vulnerabilities.
- mikeash 9y ago#4 is double-counting, since that's a special case of #3. In any case, these are all concerns for a decoder, but not for an API, which is what we're discussing here. In fact, the original comment I replied to up there was advocating the opposite: UTF-16 internally, and UTF-8 for interchange!