5 ms·

The Unicode space used to be 31-bit. UTF-8 defined 5 and 6-byte encoding sequences for this reason. When UCS-2 had to be extended to create UTF-16, the solution

The Unicode space used to be 31-bit. UTF-8 defined 5 and 6-byte encoding sequences for this reason. When UCS-2 had to be extended to create UTF-16, the solution chosen effectively limited things to a 21-bit space. Unicode was then redefined to be 21-bit, and the 5 and 6-byte encodings of UTF-8 were removed from the standard, though old systems might still accept them. It is worth noting that if we ever run out of space in 21-bit Unicode, one could conceivably re-extend it back to the original 31 bits by throwing UTF-16 under the bus. The only reason Unicode is limited to 21 bits is because of UTF-16.

teaearlgraycold 3y ago
But JS is UTF-16 native. Would it even be feasible to migrate?
- lifthrasiir 3y ago
  "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. JS implementations already do not use UTF-16 as a sole native representation anyway, so the migration should be relatively easy.
  - chrismorgan 3y ago
    > JS implementations already do not use UTF-16 as a sole native representation anyway But they still expose UTF-16 code unit semantics: U+10000–U+10FFFF come through as two UTF-16 code units, a surrogate pair. > "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. True enough. You could redefine it to have UTF-16 code unit semantics until U+10FFFF, then code point semantics beyond, maybe call this “UTF-16+ code unit semantics”. It’d definitely break a few things here and there, but then, anything would break a few things here and there, regardless of encoding type, since the range U+0000–U+10FFFF is hard-coded in so may places. Well, anything except doing surrogates again. And I’m pretty sure that’s what would be done.
    - lifthrasiir 3y ago
      Exactly this. In fact I first thought of https://ucsx.org/ https://ucsx.org/ then realized that JS strings do allow lone surrogates, and then further realized that none of this don't matter at all because the external storage format is entirely disconnected to the JS semantics. So as long as we can agree on which storage format to standardize and upgrade everything accordingly (which would be really a drama), JS wouldn't make an additional dent.
unflxw 3y ago
By throwing UTF-16 under the bus, or by extending it, in the same way that UCS-2 was created by extending UTF-16. Just like a number of UCS-2-representable codepoints in the basic plane were given up, in order to allow that number squared of non-UCS-2-representable codepoints to be representable in UTF-16, a number of UTF-16-representable codepoints in a supplemental plane could be given up, in order to allow N^2 non-UTF-16-representable codepoints in not-yet-existing supplemental planes to be representable in UTF-16-but-worse. Yes, I'm talking about putting surrogates in the surrogates.
WorldMaker 3y ago
There's also no reason to assume that if you are throwing out UTF-16 that you at that point won't also also be planning to extend past 31-bytes and out into the next plane (out into 63-bits). Assuming UTF-32 is UCS-4 and will never address future planes is how Unicode got into the situation that UTF-16 isn't infinitely expandable and is stuck at the 21-bit boundary.
- chrismorgan 3y ago
  Look, we’ve currently got about a million code points, and a trajectory and principles that should leave that easily enough for at least five thousand years, by my simplistic estimate <https://news.ycombinator.com/item?id=32286193 https://news.ycombinator.com/item?id=32286193>. There’s no point in forecasting that far in advance. And you’re concerning yourself with filling a space of two billion code points? A Unicode that would be in any danger of doing that would not be in the slightest bit recognisable as the Unicode of today. (Yeah, I know they once thought they could squeeze Unicode into 65,536 code points. I honestly don’t understand how they ever thought that would be sufficient even with Han unification, and I wish they hadn’t made that misstep that gave us the abomination that is UTF-16. But they knew running out was a hazard from the start, they just miscalculated the risk. No one that examines the matter is concerned about even the slightest risk of the current million code points being insufficient in the usefully-predictable future.)
  - WorldMaker 3y ago
    I'm just saying it is better to make no assumptions here than to make the same mistakes that led to UTF-16's accidental brokenness. There are certainly jokes that emoji's current rate of expansion could be in the billions by the end of the decade. Every time we think we have a handle on "all" of human languages we make new interesting discoveries of past languages or hidden present languages or an indigenous populace declares a need for a script they can truly call their own rather than whatever ugly mix of Latin letters some missionaries thought appropriate in the cultural insensitive past or a populace has an uprising and declares a revolutionary new start or… That's just human languages. I'm sure we'd have a ton of problems if we ever started trying to encode non-human languages. I don't think we're in danger yet of discovering that dolphins have a written script for whale languages (or wilder some sort of first contact event with non-terrestial aliens), but our science fiction loves to posit such events happening at any time and as a huge surprise. It would be great if we were at least somewhat prepared for such wild hypotheticals. Yeah, that's outside the "usefully-predictable future", but also I think Unicode has always existed in a place where the future isn't "usefully-predictable". (How many 90s Unicode architects would have predicted emoji? As only one example.)

teaearlgraycold 3y ago

But JS is UTF-16 native. Would it even be feasible to migrate?

lifthrasiir 3y ago
"UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. JS implementations already do not use UTF-16 as a sole native representation anyway, so the migration should be relatively easy.
- chrismorgan 3y ago
  > JS implementations already do not use UTF-16 as a sole native representation anyway But they still expose UTF-16 code unit semantics: U+10000–U+10FFFF come through as two UTF-16 code units, a surrogate pair. > "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. True enough. You could redefine it to have UTF-16 code unit semantics until U+10FFFF, then code point semantics beyond, maybe call this “UTF-16+ code unit semantics”. It’d definitely break a few things here and there, but then, anything would break a few things here and there, regardless of encoding type, since the range U+0000–U+10FFFF is hard-coded in so may places. Well, anything except doing surrogates again. And I’m pretty sure that’s what would be done.
  - lifthrasiir 3y ago
    Exactly this. In fact I first thought of https://ucsx.org/ https://ucsx.org/ then realized that JS strings do allow lone surrogates, and then further realized that none of this don't matter at all because the external storage format is entirely disconnected to the JS semantics. So as long as we can agree on which storage format to standardize and upgrade everything accordingly (which would be really a drama), JS wouldn't make an additional dent.

unflxw 3y ago

By throwing UTF-16 under the bus, or by extending it, in the same way that UCS-2 was created by extending UTF-16. Just like a number of UCS-2-representable codepoints in the basic plane were given up, in order to allow that number squared of non-UCS-2-representable codepoints to be representable in UTF-16, a number of UTF-16-representable codepoints in a supplemental plane could be given up, in order to allow N^2 non-UTF-16-representable codepoints in not-yet-existing supplemental planes to be representable in UTF-16-but-worse. Yes, I'm talking about putting surrogates in the surrogates.

WorldMaker 3y ago

There's also no reason to assume that if you are throwing out UTF-16 that you at that point won't also also be planning to extend past 31-bytes and out into the next plane (out into 63-bits). Assuming UTF-32 is UCS-4 and will never address future planes is how Unicode got into the situation that UTF-16 isn't infinitely expandable and is stuck at the 21-bit boundary.

chrismorgan 3y ago
Look, we’ve currently got about a million code points, and a trajectory and principles that should leave that easily enough for at least five thousand years, by my simplistic estimate <https://news.ycombinator.com/item?id=32286193 https://news.ycombinator.com/item?id=32286193>. There’s no point in forecasting that far in advance. And you’re concerning yourself with filling a space of two billion code points? A Unicode that would be in any danger of doing that would not be in the slightest bit recognisable as the Unicode of today. (Yeah, I know they once thought they could squeeze Unicode into 65,536 code points. I honestly don’t understand how they ever thought that would be sufficient even with Han unification, and I wish they hadn’t made that misstep that gave us the abomination that is UTF-16. But they knew running out was a hazard from the start, they just miscalculated the risk. No one that examines the matter is concerned about even the slightest risk of the current million code points being insufficient in the usefully-predictable future.)
- WorldMaker 3y ago
  I'm just saying it is better to make no assumptions here than to make the same mistakes that led to UTF-16's accidental brokenness. There are certainly jokes that emoji's current rate of expansion could be in the billions by the end of the decade. Every time we think we have a handle on "all" of human languages we make new interesting discoveries of past languages or hidden present languages or an indigenous populace declares a need for a script they can truly call their own rather than whatever ugly mix of Latin letters some missionaries thought appropriate in the cultural insensitive past or a populace has an uprising and declares a revolutionary new start or… That's just human languages. I'm sure we'd have a ton of problems if we ever started trying to encode non-human languages. I don't think we're in danger yet of discovering that dolphins have a written script for whale languages (or wilder some sort of first contact event with non-terrestial aliens), but our science fiction loves to posit such events happening at any time and as a huge surprise. It would be great if we were at least somewhat prepared for such wild hypotheticals. Yeah, that's outside the "usefully-predictable future", but also I think Unicode has always existed in a place where the future isn't "usefully-predictable". (How many 90s Unicode architects would have predicted emoji? As only one example.)