3 ms·

This feels less like UTF-16 being a bad format and more like JS being a bad language.

by camgunz 5y ago

Dylan16807 5y ago
Maybe. But the pattern seems to be much more common in UTF-16 systems than in UTF-8 systems.
- camgunz 5y ago
  Idk this seems vague, also it's probably pretty easy to write invalid UTF-8 (or not validate it, or get DOS'd because you panic on invalid UTF-8 in an environment variable or user input, etc.)
  - josephg 5y ago
    > it's probably pretty easy to write invalid UTF-8 In which language(s)? Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Javascript, C# and java should work like rust, since a string should always ... y'know, contain a correctly encoded string. The current situation in these languages feels like a weird middleground where String classes are actually arrays of 16 bit integers wearing a fur coat.
    - camgunz 5y ago
      > Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Oh 100%. You gotta validate every time. Again I'm not a C# or Java expert; if their stdlibs let you build invalid UTF-16 strings then that's unfortunate, but again a problem with an implementation, not an encoding. In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing.
      - josephg 5y ago
        > In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing. This is a good point. I suppose the real problem here isn't UTF-16. Its that all these languages and systems assumed unicode would never grow past 65536 characters. They weren't designed to use UTF16. They were designed to use UCS2. It was only after unicode grew past UCS2's limit that they added awful hacks to their systems in order to (badly) support UTF16. The problems I'm complaining about aren't UTF16's fault. They're problems that arise as a result of the hacks these languages added in order to support surrogate pairs. You could build a string type in rust today which used UTF16 strings internally. It would work fine, and be just as safe as the UTF8 strings in the standard library. I just really don't see the point. The verdict is in. UTF8 has won the war to be the world's interoperable format for unicode text. UTF8 is usually smaller than UTF16 (except apparently in some parts of Asia). And it doesn't need byte-order-marks or any of that guff. Its a relief that this point is basically settled. I hope that a future generation of programmers can grow up without needing to understand UCS2, UTF-16, byte-order-marks, surrogate pairs or any of this junk.
josephg 5y ago
I'm pretty sure C#, java and C++ (using UCS2 strings) all have the exact same problem.
- camgunz 5y ago
  At least Java I doubt it. char holds any valid UTF-16 value, so without casts I think it's probably pretty hard. Not an expert here though.
  - deleted 5y ago
    [deleted]
  - jcranmer 5y ago
    A Java char can hold any valid UCS-2 character, i.e. U+0000 to U+FFFF. If you need to use, say, U+1F4A9, you can't represent that with a char, so you need to use an int to hold that codepoint. A cursory glance at Java's string and character APIs reveals an awful lot of duplication between char-based "character at" and int-based "codepoint at" methods, with the former being a) broken and b) largely undeprecated despite being clearly broken.
    - camgunz 5y ago
      Yeah I think mostly you're right. The `charAt` stuff assumes UCS-2. I think the `codePointAt` stuff is fine though? Unless you think we'll get to 2 billion code points I guess (then, I've always thought Java having no unsigned types was bananas)
  - recursive 5y ago
    In a UTF-16 string, as implemented in java, you can end up with a char that represents only half a surrogate pair, that's not a full unicode scalar. In order to build the unicode scalar, you need the other half of the surrogate pair, which is another java "char".
    - camgunz 5y ago
hnfong 5y ago
You're right, but in practice most of the systems I've dealt with that claims to support "UTF-16" really means they support UCS-2, and if you need to do anything outside the BMP you can help yourself by checking surrogate characters. At least I haven't seen languages claim UTF-8 support by just giving you raw byte arrays. Before the advent of emoji (which I'm personally grateful for), the vast majority of developers aren't even aware that UCS-2 is not UTF-16 and outside the BMP Javascript's String functions are totally broken and that you actually have to check surrogates yourself to properly decode strings. And of course, it's not only Javascript... all systems/languages that mistakenly adopted UCS-2 ended up in similar places.
- camgunz 5y ago
  Yeah and I don't mean to gloss over these problems. Like, how are you supposed to realistically use UTF-16 in JS? Super clumsily apparently. IDK, again probably you can goof using UTF-8, but it does seem harder, like writing a bad byte or whatever. It would be better if string libraries actually worked though, maybe in 2023.

This feels less like UTF-16 being a bad format and more like JS being a bad language.

Dylan16807 5y ago

Maybe. But the pattern seems to be much more common in UTF-16 systems than in UTF-8 systems.

camgunz 5y ago
Idk this seems vague, also it's probably pretty easy to write invalid UTF-8 (or not validate it, or get DOS'd because you panic on invalid UTF-8 in an environment variable or user input, etc.)
- josephg 5y ago
  > it's probably pretty easy to write invalid UTF-8 In which language(s)? Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Javascript, C# and java should work like rust, since a string should always ... y'know, contain a correctly encoded string. The current situation in these languages feels like a weird middleground where String classes are actually arrays of 16 bit integers wearing a fur coat.
  - camgunz 5y ago
    > Its quite hard to write invalid UTF-8 in Rust, because the UTF8 encoding is checked when the string is created. I think this is the right choice. Oh 100%. You gotta validate every time. Again I'm not a C# or Java expert; if their stdlibs let you build invalid UTF-16 strings then that's unfortunate, but again a problem with an implementation, not an encoding. In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing.
    - josephg 5y ago
      > In other words, if Rust used UTF-16 internally it would behave exactly the same as you're describing. This is a good point. I suppose the real problem here isn't UTF-16. Its that all these languages and systems assumed unicode would never grow past 65536 characters. They weren't designed to use UTF16. They were designed to use UCS2. It was only after unicode grew past UCS2's limit that they added awful hacks to their systems in order to (badly) support UTF16. The problems I'm complaining about aren't UTF16's fault. They're problems that arise as a result of the hacks these languages added in order to support surrogate pairs. You could build a string type in rust today which used UTF16 strings internally. It would work fine, and be just as safe as the UTF8 strings in the standard library. I just really don't see the point. The verdict is in. UTF8 has won the war to be the world's interoperable format for unicode text. UTF8 is usually smaller than UTF16 (except apparently in some parts of Asia). And it doesn't need byte-order-marks or any of that guff. Its a relief that this point is basically settled. I hope that a future generation of programmers can grow up without needing to understand UCS2, UTF-16, byte-order-marks, surrogate pairs or any of this junk.

josephg 5y ago

I'm pretty sure C#, java and C++ (using UCS2 strings) all have the exact same problem.

camgunz 5y ago
At least Java I doubt it. char holds any valid UTF-16 value, so without casts I think it's probably pretty hard. Not an expert here though.
- deleted 5y ago
  [deleted]
- jcranmer 5y ago
  A Java char can hold any valid UCS-2 character, i.e. U+0000 to U+FFFF. If you need to use, say, U+1F4A9, you can't represent that with a char, so you need to use an int to hold that codepoint. A cursory glance at Java's string and character APIs reveals an awful lot of duplication between char-based "character at" and int-based "codepoint at" methods, with the former being a) broken and b) largely undeprecated despite being clearly broken.
  - camgunz 5y ago
    Yeah I think mostly you're right. The `charAt` stuff assumes UCS-2. I think the `codePointAt` stuff is fine though? Unless you think we'll get to 2 billion code points I guess (then, I've always thought Java having no unsigned types was bananas)
- recursive 5y ago
  In a UTF-16 string, as implemented in java, you can end up with a char that represents only half a surrogate pair, that's not a full unicode scalar. In order to build the unicode scalar, you need the other half of the surrogate pair, which is another java "char".
  - camgunz 5y ago

hnfong 5y ago

You're right, but in practice most of the systems I've dealt with that claims to support "UTF-16" really means they support UCS-2, and if you need to do anything outside the BMP you can help yourself by checking surrogate characters. At least I haven't seen languages claim UTF-8 support by just giving you raw byte arrays. Before the advent of emoji (which I'm personally grateful for), the vast majority of developers aren't even aware that UCS-2 is not UTF-16 and outside the BMP Javascript's String functions are totally broken and that you actually have to check surrogates yourself to properly decode strings. And of course, it's not only Javascript... all systems/languages that mistakenly adopted UCS-2 ended up in similar places.

camgunz 5y ago
Yeah and I don't mean to gloss over these problems. Like, how are you supposed to realistically use UTF-16 in JS? Super clumsily apparently. IDK, again probably you can goof using UTF-8, but it does seem harder, like writing a bad byte or whatever. It would be better if string libraries actually worked though, maybe in 2023.