7 ms·

>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever

>> UTF-8 was an amazing concept: it single handedly and brilliantly handled backward ASCII compatibility making sure that Unicode is adopted by masses. Whoever came up with it must at least receive the Nobel Peace Prize. > I'm sure Ken Thompson and Rob Pike will be happy to hear someone thinks that way. If that's the case then I think the designers of GB18030 deserve it more, because they achieved an encoding that is able to map all Unicode codepoints while being backwards compatible with GB2312, which is itself backwards compatible with ASCII. But seriously UTF-8 is like sliced bread after having dealt with we-thought-64K-is-enough-so-lets-all-use-16-bits UCS-2/UTF-16.

Dylan16807 13y ago
And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.
- millstone 13y ago
  The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem. Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.
  - spacehunt 13y ago
    UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
  - KayEss 13y ago
    >UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.
    - millstone 13y ago
      Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
  - lucian1900 13y ago
    Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.
  - gsnedders 13y ago
    This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.
    - millstone 13y ago
      Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string." But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93: If the converter encounters an ill-formed UTF-8 code unit sequence...it must not consume the successor bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute part of a well-formed UTF-8 code unit subsequence....For a UTF-8 conversion process to consume valid successor bytes is not only non-conformant, but also leaves the converter open to security exploits." "Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16. To be fair, UTF-16 has its own unique issues too, like endianness.
  - zokier 13y ago
    > The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. In what sort of situation that is a problem?
    - millstone 13y ago
      It can allow for security exploits, if the different layers of your stack handle invalid code units differently. There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

Dylan16807 13y ago

And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.

millstone 13y ago
The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem. Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.
- spacehunt 13y ago
  UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.
- KayEss 13y ago
  >UTF-16 does not suffer from that particular problem what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.
  - millstone 13y ago
    Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.
- lucian1900 13y ago
  Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.
- gsnedders 13y ago
  This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.
  - millstone 13y ago
    Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string." But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93: If the converter encounters an ill-formed UTF-8 code unit sequence...it must not consume the successor bytes as part of the ill-formed subsequence whenever those successor bytes themselves constitute part of a well-formed UTF-8 code unit subsequence....For a UTF-8 conversion process to consume valid successor bytes is not only non-conformant, but also leaves the converter open to security exploits." "Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16. To be fair, UTF-16 has its own unique issues too, like endianness.
- zokier 13y ago
  > The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. In what sort of situation that is a problem?
  - millstone 13y ago
    It can allow for security exploits, if the different layers of your stack handle invalid code units differently. There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.