5 ms·

The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format

The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format to support non-BMP code points. As a consequence, systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. > ASCII compatibility is an anti-feature. Actually, it's not. One of the main reasons it's important is... > What we need is text encoding agnosticism ... that bit that you want. If you're supporting multiple encodings, you need a reliable way of signalling that encoding. And file metadata tends to be insufficiently reliable, so it ends up being in-band for text (e.g., HTML files). If all the encodings you need to support are ASCII-compatible, and the directive to set the charset is itself pure ASCII, that means it is perfectly safe to parse the file as ASCII to figure out how to parse the file. If the charset isn't ASCII-compatible... well, that historically has led to security vulnerabilities. Ergo, a non-ASCII-compatible charset has heightened security risks. That said, my own experience when dealing with the mess that is charsets is that supporting multiple charsets is itself a painful process that isn't worth it.

josephg 5y ago
> systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. 100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode. For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string. Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16). What a mess.
- camgunz 5y ago
  This feels less like UTF-16 being a bad format and more like JS being a bad language.
  - Dylan16807 5y ago
    Maybe. But the pattern seems to be much more common in UTF-16 systems than in UTF-8 systems.
    - camgunz 5y ago
      Idk this seems vague, also it's probably pretty easy to write invalid UTF-8 (or not validate it, or get DOS'd because you panic on invalid UTF-8 in an environment variable or user input, etc.)
      - josephg 5y ago
        > it's probably pretty easy to write invalid UTF-8 In which language(s)? Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Javascript, C# and java should work like rust, since a string should always ... y'know, contain a correctly encoded string. The current situation in these languages feels like a weird middleground where String classes are actually arrays of 16 bit integers wearing a fur coat.
        camgunz 5y ago
        > Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Oh 100%. You gotta validate every time. Again I'm not a C# or Java expert; if their stdlibs let you build invalid UTF-16 strings then that's unfortunate, but again a problem with an implementation, not an encoding. In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing.
        5y ago
camgunz 5y ago
Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including ASCII suddenly having UTF-8 bytes in it). As soon as you're in a "assume an encoding and maybe be surprised", you're asking to be surprised. Totally agree these discussions circle around to "you need some kind of OOB way to know what encoding you're dealing with", which puts us back to everything being encoding agnostic, with some way to configure it. Maybe that's locales, maybe that's a type 32-byte integer at the beginning, dunno.
- kevin_thibedeau 5y ago
  > including ASCII suddenly having UTF-8 bytes People have been dealing with 8-bit ASCII extensions for longer than Unicode has been around. It isn't overwhelmingly difficult to have 8-bit clean text handling without any awareness of the encoding.
- jcranmer 5y ago
  No, the best way is not to have people specify what encoding they intend to use--it is instead to not give people a choice. When you give people a choice, given that one encoding is far dominant above all others, many people will start assuming that is the only viable choice regardless of any actual declarations to the contrary. An example of the latter effect in action is the system charset of Linux and most other Unixen... in practice, virtually every system is using UTF-8 for things like pathnames and the like. However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. Of course, there's a few more advantages to always using UTF-8: * You eliminate the code path of "I'm sorry, I can't save the text that you wrote," since every character is usable in UTF-8, which is a feature very rare outside of UTF-* charsets. And, honestly, that's a wonderful code path to eliminate. * One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. This allows you to build systems that produce errors on bad input, which means that maybe, just maybe, the people who produced the broken input will discover their problem and fix it.
  - camgunz 5y ago

josephg 5y ago

> systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible. 100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode. For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string. Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16). What a mess.

camgunz 5y ago
This feels less like UTF-16 being a bad format and more like JS being a bad language.
- Dylan16807 5y ago
  Maybe. But the pattern seems to be much more common in UTF-16 systems than in UTF-8 systems.
  - camgunz 5y ago
    Idk this seems vague, also it's probably pretty easy to write invalid UTF-8 (or not validate it, or get DOS'd because you panic on invalid UTF-8 in an environment variable or user input, etc.)
    - josephg 5y ago
      > it's probably pretty easy to write invalid UTF-8 In which language(s)? Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Javascript, C# and java should work like rust, since a string should always ... y'know, contain a correctly encoded string. The current situation in these languages feels like a weird middleground where String classes are actually arrays of 16 bit integers wearing a fur coat.
      - camgunz 5y ago
        > Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Oh 100%. You gotta validate every time. Again I'm not a C# or Java expert; if their stdlibs let you build invalid UTF-16 strings then that's unfortunate, but again a problem with an implementation, not an encoding. In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing.
        5y ago

camgunz 5y ago

Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including ASCII suddenly having UTF-8 bytes in it). As soon as you're in a "assume an encoding and maybe be surprised", you're asking to be surprised. Totally agree these discussions circle around to "you need some kind of OOB way to know what encoding you're dealing with", which puts us back to everything being encoding agnostic, with some way to configure it. Maybe that's locales, maybe that's a type 32-byte integer at the beginning, dunno.

kevin_thibedeau 5y ago
> including ASCII suddenly having UTF-8 bytes People have been dealing with 8-bit ASCII extensions for longer than Unicode has been around. It isn't overwhelmingly difficult to have 8-bit clean text handling without any awareness of the encoding.
jcranmer 5y ago
No, the best way is not to have people specify what encoding they intend to use--it is instead to not give people a choice. When you give people a choice, given that one encoding is far dominant above all others, many people will start assuming that is the only viable choice regardless of any actual declarations to the contrary. An example of the latter effect in action is the system charset of Linux and most other Unixen... in practice, virtually every system is using UTF-8 for things like pathnames and the like. However, the mechanism to indicate that this is the case is frequently enough unreliable that it's often less bug-prone to assume the system is UTF-8 anyways. Of course, there's a few more advantages to always using UTF-8: * You eliminate the code path of "I'm sorry, I can't save the text that you wrote," since every character is usable in UTF-8, which is a feature very rare outside of UTF-* charsets. And, honestly, that's a wonderful code path to eliminate. * One of the features of UTF-8 is that virtually any text that isn't UTF-8 will fail to parse as UTF-8. This allows you to build systems that produce errors on bad input, which means that maybe, just maybe, the people who produced the broken input will discover their problem and fix it.
- camgunz 5y ago