4 ms·

> just an internal representation for a few older toolkits and programming languages Maybe "older" but very much current, Qt's QString type internally uses UTF

> just an internal representation for a few older toolkits and programming languages Maybe "older" but very much current, Qt's QString type internally uses UTF-16. https://doc.qt.io/qt-5/qstring.html https://doc.qt.io/qt-5/qstring.html

lokedhs 5y ago
And both Java and Javascript, an thus any language derived from from them. I'd go as far as saying that a majority of the code being written today uses UTF-16. With these languages not even providing alternative API's that are easy to use means it'll be some time before we don't have to suffer this.
- qalmakka 5y ago
  I've seen a trend from those languages and APIs of hiding UTF-16 away. They have to interoperate with everything else using just UTF8, while the rest of the world doesn't give a half damn about UTF-16 (the fact you have to also take endianess into account I think was one of the biggest crippling blows 16 bit encoding had ever received)
qalmakka 5y ago
Yeah, because Qt started in the early 90s, when UCS2 hadn't still fiascoed and it was being adopted in droves as the "newer" solution to all encoding issues - you simply swapped `char` for `wchar_t`, made a `w` version of every C and C++ IO function and that's it, right? Sadly, it wasn't it. 16 bits weren't enough, and stuff like the fact a Unicode rune ≠ printed character (i.e. [è] can either be a single codepoints, or a combination of a [`] modifier and the latin letter [e]) meant there was basically no point in using 16/32 bit chars in the first place. When people really understood this it was almost the '00s, and stuff like Python, Windows, macOS (due to NextStep), Java, .NET, Qt were stuck. It's impossible to go back to plain `char` without annihilating backward compatibility, so everyone kept using it internally. Fun fact, some of those languages and frameworks I mentioned never bothered switching completely to UTF-16 - for instance, Python now uses a weird mixture of ASCII and UCS2 internally.
- rav 5y ago
  > for instance, Python now uses a weird mixture of ASCII and UCS2 internally. Really? Last I heard (PEP 393), the rule was: "8 bits if all codepoints are less than 256 (i.e. Latin-1); 16 bits if all codepoints are less than 2^16 (i.e. BMP); otherwise 32 bits". This means that text with all Latin-1 characters (which are approximately the first 256 codepoints of Unicode) will be stored internally as, well, Latin-1. This implies that ASCII strings are stored as ASCII.
  - qalmakka 5y ago
    Yep, that's what I meant. They either use ASCII, UCS2 or UCS4 depending on the type of string. Doesn't make a lot of sense to me, but I guess they couldn't just throw 16 bit chars away.
    - chronial 5y ago
      I'm not 100% sure on this, but I don't backwards-compatibility mattered in this decision. They wanted memory-efficient and O(1)-indexable strings.

lokedhs 5y ago

And both Java and Javascript, an thus any language derived from from them. I'd go as far as saying that a majority of the code being written today uses UTF-16. With these languages not even providing alternative API's that are easy to use means it'll be some time before we don't have to suffer this.

qalmakka 5y ago
I've seen a trend from those languages and APIs of hiding UTF-16 away. They have to interoperate with everything else using just UTF8, while the rest of the world doesn't give a half damn about UTF-16 (the fact you have to also take endianess into account I think was one of the biggest crippling blows 16 bit encoding had ever received)

qalmakka 5y ago

Yeah, because Qt started in the early 90s, when UCS2 hadn't still fiascoed and it was being adopted in droves as the "newer" solution to all encoding issues - you simply swapped `char` for `wchar_t`, made a `w` version of every C and C++ IO function and that's it, right? Sadly, it wasn't it. 16 bits weren't enough, and stuff like the fact a Unicode rune ≠ printed character (i.e. [è] can either be a single codepoints, or a combination of a [`] modifier and the latin letter [e]) meant there was basically no point in using 16/32 bit chars in the first place. When people really understood this it was almost the '00s, and stuff like Python, Windows, macOS (due to NextStep), Java, .NET, Qt were stuck. It's impossible to go back to plain `char` without annihilating backward compatibility, so everyone kept using it internally. Fun fact, some of those languages and frameworks I mentioned never bothered switching completely to UTF-16 - for instance, Python now uses a weird mixture of ASCII and UCS2 internally.

rav 5y ago
> for instance, Python now uses a weird mixture of ASCII and UCS2 internally. Really? Last I heard (PEP 393), the rule was: "8 bits if all codepoints are less than 256 (i.e. Latin-1); 16 bits if all codepoints are less than 2^16 (i.e. BMP); otherwise 32 bits". This means that text with all Latin-1 characters (which are approximately the first 256 codepoints of Unicode) will be stored internally as, well, Latin-1. This implies that ASCII strings are stored as ASCII.
- qalmakka 5y ago
  Yep, that's what I meant. They either use ASCII, UCS2 or UCS4 depending on the type of string. Doesn't make a lot of sense to me, but I guess they couldn't just throw 16 bit chars away.
  - chronial 5y ago
    I'm not 100% sure on this, but I don't backwards-compatibility mattered in this decision. They wanted memory-efficient and O(1)-indexable strings.