2 ms·
If you truncate a UTF-8 string mid-character, then can’t that sequence swallow the next byte, such as <, which would break XML parsing completely?
by JimDabell 5y ago
If you truncate a UTF-8 string mid-character, then can’t that sequence swallow the next byte, such as <, which would break XML parsing completely?
- toast0 5y agoNo, UTF-8 is self synchronizing... The first byte of a code point is clearly the first byte of a code point, so it's possible to tell that there was a truncated code point, as long as the next one starts correctly (if you lost two bytes in a row, you might get a mashed up code point, but that's a different loss scenario)
- _moof 5y agoUTF-8 encoding works like so: 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx ...where the "x" bits encode the code point. So if you're halfway through a four-byte sequence and see a byte like 0xxxxxxx, you know something's wrong.