5 ms·

> if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Paradoxically, trying to do "the right thing" and b

> if you have been handling Unicode and using wide characters, you have not been handling Unicode properly. Paradoxically, trying to do "the right thing" and being an "early adopter" of (the now called) UCS-2 was a "mistake", as both Java and Windows can attest, by getting "stuck" supporting the worst possible Unicode encoding ad-infinitum. UTF-8 is the "obviously correct" choice (from the hindsight afforded by us talking about this in 2021). I still find it funny that emojis of all things are what actually got the anglosphere to actually write software that isn't completely broken for the other 5.5 billion people out there.

lenkite 6y ago
Java hasn't been UCS-2 since Java 5.0 - nearly a decade.
- deleted 6y ago
  [deleted]
RcouF1uZ4gsC 6y ago
> actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2. So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.
- Quekid5 6y ago
  Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)
  - tsimionescu 6y ago
    Even with proper UTF-8, you'll get situations where you can insert a space in the middle of an emoji and split it into other emoji + unprintable characters. The encoding is irrelevant, you need proper Unicode support to avoid these problems.
    - account42 6y ago
      Unfortunately proper unicode support for many operations means carrying around tons of data since important properties of code points cannot be derived from the codepoints themselves.
      - tsimionescu 6y ago
        Exactly. UTF-8 doesn't and cant fix this problem, you need a full Unicode library if you want to correctly handle human text. If you don't, why bother with UTF-8 instead of something simpler, like ASCII?
        account42 6y ago
        > If you don't, why bother with UTF-8 instead of something simpler, like ASCII? Because unicode support isn't binary. Being able to pass along and not mange blobs of unicode is already a lot better than ASCII-only.
- estebank 6y ago
  Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. For multiple reasons, Unicode has multiple representations for the same glyph[1], so if you have ü, it can either be a the two bytes 00FC[2], or u followed by the ¨ diacritic U+0308. If you don't account for this things like "reversing a string" or "give me a substring" or "how long is the string on screen" will be subtly buggy. If you handle UCS-2 correctly, then it's fine and a reasonable technical limitation, but the emphasis of that sentence is on correctly. [1]: https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence [2]: https://www.compart.com/en/unicode/U+00FC https://www.compart.com/en/unicode/U+00FC
  - Dylan16807 6y ago
    > Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. It can be. But critically for this conversation, fixing your code to support emoji and other non-BMP characters doesn't necessarily fix those problems.
    - account42 6y ago
      Except it does since a single emoji can also be made up of multiple code points.
      - Dylan16807 6y ago
        But that only causes subtle wrongness, which can continue to go unnoticed.
- rectang 6y ago
  > So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people. One of the great things about using Rust is that I don't have to have this argument. There doesn't have to be a debate about whether we should invest in fixing subtly broken code. Rust string-handling which generally works for Europeans will also work for Chinese! This isn't the fault of the languages which were designed for UCS-2 (then known just as "Unicode"). But the fact that Rust emerged after UTF-8's ascendance means that Rust's users mostly get to avoid the UCS-2/UTF-16 legacy tarpit.
  - tsimionescu 6y ago
    > Rust string-handling which generally works for Europeans will also work for Chinese! I doubt that is any more true than Java. You can easily write code in Rust that assumes you can split a String anywhere and get two valid strings, that you can compute the length of a String and get information about how long the printed representation will be, that you can find a substring in that string by simply iterating through UTF-8 code points etc. All of these assumptions are about as wrong in UTF-8 Rust as they are in UTF-16 Java.
  - deleted 6y ago
    [deleted]
TwoBit 6y ago
Well if memory cost wasn't an issue then UCS-4 would be nicest. But 4x the memory for most strings is currently unacceptable.
- ohazi 6y ago
  There are a lot of instances where I wouldn't mind the memory cost, but would very much mind not automatically being compatible with ascii strings.
- lokedhs 6y ago
  That is true, but the benefits of UTF-32 is minor compared to UTF-8, and might not be worth the cost. And I say that as someone who is developing a language which only has UTF-32 support. The problem is that even with UTF-32, doing things like splitting strings is inherently unsafe, so you are still going to need a Unicode library to do proper splitting by grapheme cluster. In practice, almost all string splitting works on ASCII text, and assumes everything else is data that should not be manipulated. For this, UTF-8 is perfectly acceptable.
- fanf2 6y ago
  UCS-4 is still a variable-length encoding, because various accented characters and emoji use multiple code points. One advantage of UTF-8 is that it makes you confront variable-length characters head on.
mjevans 6y ago
Correction to the date and timeline. UTF-8 has been the correct choice since: " >From ken Tue Sep 8 03:22:07 EDT 1992 " As discussed last week in the quoted history https://news.ycombinator.com/item?id=26735958 https://news.ycombinator.com/item?id=26735958 of UTF-8's mail messages: https://doc.cat-v.org/bell_labs/utf-8_history https://doc.cat-v.org/bell_labs/utf-8_history It appears that Unicode arose between 1990 and 1991 for the initial versions https://en.wikipedia.org/wiki/Universal_Coded_Character_Set#History https://en.wikipedia.org/wiki/Universal_Coded_Character_Set#... with the first published version of the specification in 1993: "ISO/IEC 10646-1:1993 = Unicode 1.1" As discussed another time on Hacker News https://news.ycombinator.com/item?id=20600195 https://news.ycombinator.com/item?id=20600195 https://unascribed.com/b/2019-08-02-the-tragedy-of-ucs2.html https://unascribed.com/b/2019-08-02-the-tragedy-of-ucs2.html It was around 1996 when it became clearly obvious (software got shipped to end users who cared and they complained back) that UCS-2 (16 bit characters) would be insufficient. +++ Pragmatically this is forever, as long as backwards compatibility must be maintained, the exiting APIs which are built around the crazy 16 bit standard need to exist; but there's little reason they have to be native, rather than wrappers for UTF-8 compatible APIs. It would even be a good time to standardize on a single user space programming API and have implementations on every operating system. Preferably including basic drawing and font layout functions. So that finally, most programs could be written once, compiled on a platform of choice, and work.

lenkite 6y ago

Java hasn't been UCS-2 since Java 5.0 - nearly a decade.

deleted 6y ago
[deleted]

RcouF1uZ4gsC 6y ago

> actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2. So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.

Quekid5 6y ago
Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)
- tsimionescu 6y ago
  Even with proper UTF-8, you'll get situations where you can insert a space in the middle of an emoji and split it into other emoji + unprintable characters. The encoding is irrelevant, you need proper Unicode support to avoid these problems.
  - account42 6y ago
    Unfortunately proper unicode support for many operations means carrying around tons of data since important properties of code points cannot be derived from the codepoints themselves.
    - tsimionescu 6y ago
      Exactly. UTF-8 doesn't and cant fix this problem, you need a full Unicode library if you want to correctly handle human text. If you don't, why bother with UTF-8 instead of something simpler, like ASCII?
      - account42 6y ago
        > If you don't, why bother with UTF-8 instead of something simpler, like ASCII? Because unicode support isn't binary. Being able to pass along and not mange blobs of unicode is already a lot better than ASCII-only.
estebank 6y ago
Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. For multiple reasons, Unicode has multiple representations for the same glyph[1], so if you have ü, it can either be a the two bytes 00FC[2], or u followed by the ¨ diacritic U+0308. If you don't account for this things like "reversing a string" or "give me a substring" or "how long is the string on screen" will be subtly buggy. If you handle UCS-2 correctly, then it's fine and a reasonable technical limitation, but the emphasis of that sentence is on correctly. [1]: https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence [2]: https://www.compart.com/en/unicode/U+00FC https://www.compart.com/en/unicode/U+00FC
- Dylan16807 6y ago
  > Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. It can be. But critically for this conversation, fixing your code to support emoji and other non-BMP characters doesn't necessarily fix those problems.
  - account42 6y ago
    Except it does since a single emoji can also be made up of multiple code points.
    - Dylan16807 6y ago
      But that only causes subtle wrongness, which can continue to go unnoticed.
rectang 6y ago
> So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people. One of the great things about using Rust is that I don't have to have this argument. There doesn't have to be a debate about whether we should invest in fixing subtly broken code. Rust string-handling which generally works for Europeans will also work for Chinese! This isn't the fault of the languages which were designed for UCS-2 (then known just as "Unicode"). But the fact that Rust emerged after UTF-8's ascendance means that Rust's users mostly get to avoid the UCS-2/UTF-16 legacy tarpit.
- tsimionescu 6y ago
  > Rust string-handling which generally works for Europeans will also work for Chinese! I doubt that is any more true than Java. You can easily write code in Rust that assumes you can split a String anywhere and get two valid strings, that you can compute the length of a String and get information about how long the printed representation will be, that you can find a substring in that string by simply iterating through UTF-8 code points etc. All of these assumptions are about as wrong in UTF-8 Rust as they are in UTF-16 Java.
- deleted 6y ago
  [deleted]

TwoBit 6y ago

Well if memory cost wasn't an issue then UCS-4 would be nicest. But 4x the memory for most strings is currently unacceptable.

ohazi 6y ago
There are a lot of instances where I wouldn't mind the memory cost, but would very much mind not automatically being compatible with ascii strings.
lokedhs 6y ago
That is true, but the benefits of UTF-32 is minor compared to UTF-8, and might not be worth the cost. And I say that as someone who is developing a language which only has UTF-32 support. The problem is that even with UTF-32, doing things like splitting strings is inherently unsafe, so you are still going to need a Unicode library to do proper splitting by grapheme cluster. In practice, almost all string splitting works on ASCII text, and assumes everything else is data that should not be manipulated. For this, UTF-8 is perfectly acceptable.
fanf2 6y ago
UCS-4 is still a variable-length encoding, because various accented characters and emoji use multiple code points. One advantage of UTF-8 is that it makes you confront variable-length characters head on.

mjevans 6y ago

Correction to the date and timeline. UTF-8 has been the correct choice since: " >From ken Tue Sep 8 03:22:07 EDT 1992 " As discussed last week in the quoted history https://news.ycombinator.com/item?id=26735958 https://news.ycombinator.com/item?id=26735958 of UTF-8's mail messages: https://doc.cat-v.org/bell_labs/utf-8_history https://doc.cat-v.org/bell_labs/utf-8_history It appears that Unicode arose between 1990 and 1991 for the initial versions https://en.wikipedia.org/wiki/Universal_Coded_Character_Set#History https://en.wikipedia.org/wiki/Universal_Coded_Character_Set#... with the first published version of the specification in 1993: "ISO/IEC 10646-1:1993 = Unicode 1.1" As discussed another time on Hacker News https://news.ycombinator.com/item?id=20600195 https://news.ycombinator.com/item?id=20600195 https://unascribed.com/b/2019-08-02-the-tragedy-of-ucs2.html https://unascribed.com/b/2019-08-02-the-tragedy-of-ucs2.html It was around 1996 when it became clearly obvious (software got shipped to end users who cared and they complained back) that UCS-2 (16 bit characters) would be insufficient. +++ Pragmatically this is forever, as long as backwards compatibility must be maintained, the exiting APIs which are built around the crazy 16 bit standard need to exist; but there's little reason they have to be native, rather than wrappers for UTF-8 compatible APIs. It would even be a good time to standardize on a single user space programming API and have implementations on every operating system. Preferably including basic drawing and font layout functions. So that finally, most programs could be written once, compiled on a platform of choice, and work.