5 ms·

> actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages t

> actually write software that isn't completely broken for the other 5.5 billion people out there I thought that Chinese and Japanese are the only languages that UCS-2 has trouble fully representing. I believe all the other living languages can actually be represented by UCS-2. So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people.

Quekid5 6y ago
Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)
- tsimionescu 6y ago
  Even with proper UTF-8, you'll get situations where you can insert a space in the middle of an emoji and split it into other emoji + unprintable characters. The encoding is irrelevant, you need proper Unicode support to avoid these problems.
  - account42 6y ago
    Unfortunately proper unicode support for many operations means carrying around tons of data since important properties of code points cannot be derived from the codepoints themselves.
    - tsimionescu 6y ago
      Exactly. UTF-8 doesn't and cant fix this problem, you need a full Unicode library if you want to correctly handle human text. If you don't, why bother with UTF-8 instead of something simpler, like ASCII?
      - account42 6y ago
        > If you don't, why bother with UTF-8 instead of something simpler, like ASCII? Because unicode support isn't binary. Being able to pass along and not mange blobs of unicode is already a lot better than ASCII-only.
estebank 6y ago
Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. For multiple reasons, Unicode has multiple representations for the same glyph[1], so if you have ü, it can either be a the two bytes 00FC[2], or u followed by the ¨ diacritic U+0308. If you don't account for this things like "reversing a string" or "give me a substring" or "how long is the string on screen" will be subtly buggy. If you handle UCS-2 correctly, then it's fine and a reasonable technical limitation, but the emphasis of that sentence is on correctly. [1]: https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence [2]: https://www.compart.com/en/unicode/U+00FC https://www.compart.com/en/unicode/U+00FC
- Dylan16807 6y ago
  > Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. It can be. But critically for this conversation, fixing your code to support emoji and other non-BMP characters doesn't necessarily fix those problems.
  - account42 6y ago
    Except it does since a single emoji can also be made up of multiple code points.
    - Dylan16807 6y ago
      But that only causes subtle wrongness, which can continue to go unnoticed.
rectang 6y ago
> So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people. One of the great things about using Rust is that I don't have to have this argument. There doesn't have to be a debate about whether we should invest in fixing subtly broken code. Rust string-handling which generally works for Europeans will also work for Chinese! This isn't the fault of the languages which were designed for UCS-2 (then known just as "Unicode"). But the fact that Rust emerged after UTF-8's ascendance means that Rust's users mostly get to avoid the UCS-2/UTF-16 legacy tarpit.
- tsimionescu 6y ago
  > Rust string-handling which generally works for Europeans will also work for Chinese! I doubt that is any more true than Java. You can easily write code in Rust that assumes you can split a String anywhere and get two valid strings, that you can compute the length of a String and get information about how long the printed representation will be, that you can find a substring in that string by simply iterating through UTF-8 code points etc. All of these assumptions are about as wrong in UTF-8 Rust as they are in UTF-16 Java.
- deleted 6y ago
  [deleted]

Quekid5 6y ago

Do we not consider emoji a language unto itself? I was actually quite bemused to discover that some code review software I was using allowed me to "cursor" halfway through a smiley face emoji and enter a space (typing too fast to pay attention)... causing the infamous "box characters" because I'd accidentally split the smiley down the middle. I get the need for extreme backward compat in browsers, but... this seems like one of those things that just might be worth fixing. Maybe a "use utf8" directive? :)

tsimionescu 6y ago
Even with proper UTF-8, you'll get situations where you can insert a space in the middle of an emoji and split it into other emoji + unprintable characters. The encoding is irrelevant, you need proper Unicode support to avoid these problems.
- account42 6y ago
  Unfortunately proper unicode support for many operations means carrying around tons of data since important properties of code points cannot be derived from the codepoints themselves.
  - tsimionescu 6y ago
    Exactly. UTF-8 doesn't and cant fix this problem, you need a full Unicode library if you want to correctly handle human text. If you don't, why bother with UTF-8 instead of something simpler, like ASCII?
    - account42 6y ago
      > If you don't, why bother with UTF-8 instead of something simpler, like ASCII? Because unicode support isn't binary. Being able to pass along and not mange blobs of unicode is already a lot better than ASCII-only.

estebank 6y ago

Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. For multiple reasons, Unicode has multiple representations for the same glyph[1], so if you have ü, it can either be a the two bytes 00FC[2], or u followed by the ¨ diacritic U+0308. If you don't account for this things like "reversing a string" or "give me a substring" or "how long is the string on screen" will be subtly buggy. If you handle UCS-2 correctly, then it's fine and a reasonable technical limitation, but the emphasis of that sentence is on correctly. [1]: https://en.wikipedia.org/wiki/Unicode_equivalence https://en.wikipedia.org/wiki/Unicode_equivalence [2]: https://www.compart.com/en/unicode/U+00FC https://www.compart.com/en/unicode/U+00FC

Dylan16807 6y ago
> Code that treats UCS-2 as "wide ASCII" can be subtly wrong even for western european languages. It can be. But critically for this conversation, fixing your code to support emoji and other non-BMP characters doesn't necessarily fix those problems.
- account42 6y ago
  Except it does since a single emoji can also be made up of multiple code points.
  - Dylan16807 6y ago
    But that only causes subtle wrongness, which can continue to go unnoticed.

rectang 6y ago

> So using UCS-2 would actually work for almost everyone except maybe 1.5 billion people. One of the great things about using Rust is that I don't have to have this argument. There doesn't have to be a debate about whether we should invest in fixing subtly broken code. Rust string-handling which generally works for Europeans will also work for Chinese! This isn't the fault of the languages which were designed for UCS-2 (then known just as "Unicode"). But the fact that Rust emerged after UTF-8's ascendance means that Rust's users mostly get to avoid the UCS-2/UTF-16 legacy tarpit.

tsimionescu 6y ago
> Rust string-handling which generally works for Europeans will also work for Chinese! I doubt that is any more true than Java. You can easily write code in Rust that assumes you can split a String anywhere and get two valid strings, that you can compute the length of a String and get information about how long the printed representation will be, that you can find a substring in that string by simply iterating through UTF-8 code points etc. All of these assumptions are about as wrong in UTF-8 Rust as they are in UTF-16 Java.
deleted 6y ago
[deleted]