4 ms·

The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot b

The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem. Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.

spacehunt 13y ago
UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
KayEss 13y ago
>UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.
- millstone 13y ago
  Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
lucian1900 13y ago
Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.
gsnedders 13y ago
This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.
- millstone 13y ago
  Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string." But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93: If the converter encounters an ill-formed UTF-8 code unit sequence...it must not consume the successor bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute part of a well-formed UTF-8 code unit subsequence....For a UTF-8 conversion process to consume valid successor bytes is not only non-conformant, but also leaves the converter open to security exploits." "Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16. To be fair, UTF-16 has its own unique issues too, like endianness.
zokier 13y ago
> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. In what sort of situation that is a problem?
- millstone 13y ago
  It can allow for security exploits, if the different layers of your stack handle invalid code units differently. There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

spacehunt 13y ago

UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.

KayEss 13y ago

>UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.

millstone 13y ago
Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.

lucian1900 13y ago

Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.

gsnedders 13y ago

This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.

millstone 13y ago
Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string." But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93: If the converter encounters an ill-formed UTF-8 code unit sequence...it must not consume the successor bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute part of a well-formed UTF-8 code unit subsequence....For a UTF-8 conversion process to consume valid successor bytes is not only non-conformant, but also leaves the converter open to security exploits." "Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16. To be fair, UTF-16 has its own unique issues too, like endianness.

zokier 13y ago

> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. In what sort of situation that is a problem?

millstone 13y ago
It can allow for security exploits, if the different layers of your stack handle invalid code units differently. There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.