3 ms·

The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as

The reasoning behind using UTF-16/UCS-2 is that then you can plug your ears and treat 1 char == 1 user-visible glyph on the screen, so programmers that acted as if ASCII was the only encoding in existence could continue treating strings in the same way (using their length to calculate their user-visible length, indexing directly on specific characters to change them, etc). All of those practices are immediately wrong once UTF-32 came in existence and UTF-16 became a variable length encoding. But even if that hadn't happened, what you want to be operating on is not characters, but grapheme clusters, which are equivalent to a vector of chars. Otherwise you won't handle the distinction between ë and ë or emojis correctly.

magicalhippo 6y ago
But how is that different from the underlying encoding being UTF-8? edit: For example, we do a lot of string manipulation in Delphi. We might split a string in multiple pieces and glue them together again somehow. But our separators are fixed, say a tab character, or a semicolon. So this stiching and joining is oblivious to whatever emojis and other funky stuff that might be inbetween. How is this doing it wrong? I mean yea sure you CAN screw it up by individually manipulating characters. But I don't see how an UTF-8 encoded string in itself prevents you from doing the same kind of mistakes.
- josephg 6y ago
  Splitting and glueing is fine. But imagine 3 systems: system A is obviously wrong. It crashes on any input. System B is subtly wrong. It works most of the time, but you’re getting reports that it crashes if you input Korean characters and you don’t know Korean or how to type those characters. System C is correct. Obviously C is better than A or B, because you want people to have a good experience with your software. But weirdly, system A (broken always) is usually better than system B (broken in weird hard to test ways). The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Code that is broken in subtle ways will get shipped and cause user frustration, churn, support calls, and so on. The problem with UCS-2 is it falls into system B. It works most of the time, for all the languages I can type. It breaks with some inputs I can’t type on my keyboard. So the bugs make it through to production. UTF-8 is more like system A than system B. You get multibyte code sequences as soon as you leave ASCII, so it’s easier to break. (Though it really took emoji for people to be serious about making everything work.)
  - magicalhippo 6y ago
    Right, but the claim was this: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly I agree that UTF-8 is a better encoding overall for the majority of cases. I don't think that means UTF-16, which for example Delphi UnicodeStrings are[1], is not proper. edit: maybe this is a language confusion thing. For historically tragic reasons, we're stuck with "char" as the basic element of string types in lots of languages. In Delphi a "widechar" is technically a code unit[2], and may or may not represent a code point. This is how I interpreted the OP. Maybe he meant wide characters as code points, in which I would agree. [1]: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_in_RAD_Studio#New_String_Type:_UnicodeString http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_i... [2]: https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
    - josephg 6y ago
      Yeah I hear you. Its definitely possible to write correct code using UCS-2 (where each "char" sometimes represents only half of a codepoint). But its easy to end up with subtly broken code, that only breaks for non-english speakers who don't know enough english to file a bug report. The ergonomics of the language guide you in that direction when, as you say, a "char" doesn't actually represent a character. Or even an atomic unicode codepoint. And when string.length gives you an essentially meaningless value. Luckily, code like this will also break when encountering emoji. Thats great, because it means my local users will complain about these bugs and they're easy for me to reproduce. As a result these problems are slowly being fixed.
  - dragonwriter 6y ago
    > The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Having worked in tech support for a piece of very expensive (~$100k per install annual support/license fee in the late 90s) enterprise software that had a GA release shipped to customers with a syntax error in an install script, I would state that more like “code that’s non-subtly broken is less likely be shipped to customers before it works.”

magicalhippo 6y ago

But how is that different from the underlying encoding being UTF-8? edit: For example, we do a lot of string manipulation in Delphi. We might split a string in multiple pieces and glue them together again somehow. But our separators are fixed, say a tab character, or a semicolon. So this stiching and joining is oblivious to whatever emojis and other funky stuff that might be inbetween. How is this doing it wrong? I mean yea sure you CAN screw it up by individually manipulating characters. But I don't see how an UTF-8 encoded string in itself prevents you from doing the same kind of mistakes.

josephg 6y ago
Splitting and glueing is fine. But imagine 3 systems: system A is obviously wrong. It crashes on any input. System B is subtly wrong. It works most of the time, but you’re getting reports that it crashes if you input Korean characters and you don’t know Korean or how to type those characters. System C is correct. Obviously C is better than A or B, because you want people to have a good experience with your software. But weirdly, system A (broken always) is usually better than system B (broken in weird hard to test ways). The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Code that is broken in subtle ways will get shipped and cause user frustration, churn, support calls, and so on. The problem with UCS-2 is it falls into system B. It works most of the time, for all the languages I can type. It breaks with some inputs I can’t type on my keyboard. So the bugs make it through to production. UTF-8 is more like system A than system B. You get multibyte code sequences as soon as you leave ASCII, so it’s easier to break. (Though it really took emoji for people to be serious about making everything work.)
- magicalhippo 6y ago
  Right, but the claim was this: if you have been handling Unicode and using wide characters, you have not been handling Unicode properly I agree that UTF-8 is a better encoding overall for the majority of cases. I don't think that means UTF-16, which for example Delphi UnicodeStrings are[1], is not proper. edit: maybe this is a language confusion thing. For historically tragic reasons, we're stuck with "char" as the basic element of string types in lots of languages. In Delphi a "widechar" is technically a code unit[2], and may or may not represent a code point. This is how I interpreted the OP. Maybe he meant wide characters as code points, in which I would agree. [1]: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_in_RAD_Studio#New_String_Type:_UnicodeString http://docwiki.embarcadero.com/RADStudio/Sydney/en/Unicode_i... [2]: https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
  - josephg 6y ago
    Yeah I hear you. Its definitely possible to write correct code using UCS-2 (where each "char" sometimes represents only half of a codepoint). But its easy to end up with subtly broken code, that only breaks for non-english speakers who don't know enough english to file a bug report. The ergonomics of the language guide you in that direction when, as you say, a "char" doesn't actually represent a character. Or even an atomic unicode codepoint. And when string.length gives you an essentially meaningless value. Luckily, code like this will also break when encountering emoji. Thats great, because it means my local users will complain about these bugs and they're easy for me to reproduce. As a result these problems are slowly being fixed.
- dragonwriter 6y ago
  > The reason is that code that’s broken can be easily debugged and fixed, and will not be shipped to customers until it works. Having worked in tech support for a piece of very expensive (~$100k per install annual support/license fee in the late 90s) enterprise software that had a GA release shipped to customers with a syntax error in an install script, I would state that more like “code that’s non-subtly broken is less likely be shipped to customers before it works.”