5 ms·

I expect UTF-16 to die eventually and let us have more than 21 bits of codepoints, but we'll never ever need even 30 bits, let alone 32 bits, of codepoints. I

I expect UTF-16 to die eventually and let us have more than 21 bits of codepoints, but we'll never ever need even 30 bits, let alone 32 bits, of codepoints. I have to imagine that you're just jesting :0

WorldMaker 8y ago
It's not entirely a jest, but it is funny to think about. The question will be then, as it has been every time before: what do we mean by "Universal" in Unicode? One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. Less of a jest is that there's always more symbols to encode. We've been a wonderfully creative species, and we're still rediscovering old written languages and discovering new bits of physics and math that can benefit from new symbols. We've got tons of interesting fictional language encodings to consider that are currently not likely to be encoded in Unicode today because they are locked in that struggle between the needs of Academia, the interests the Enthusiasts, and the money/complications of Intellectual Property laws. (Klingon, Tolkien's Elvish languages, D'ni, Timelord Sigils, etc.) But if those seesaws tip towards enthusiasts or academia that changes quickly. Meanwhile, creative artists keep inventing new glyphs and languages to encode. That's just our one species on this planet. What if we discover more or get better at encoding existing non-human languages? Then we really get to test that "Universal" bit in the name, huh? (That's also obviously a bit of jest, but an interesting sci-fi hope/yearning there, too. It would be nice to encode some non-human languages for a change.) So, all of the above, based on current assumptions (emojis will slow down, IP law remains vicious, Fermi's Paradox won't been solved in our lifetimes, etc) may still be unlikely to push us past the Astral Plane, but that doesn't make it silly to assume that we may never need to open the next plane.
- cryptonector 8y ago
  There's always more glyphs to encode, but I think it will be a long time before we reach hundreds of millions of glyphs, even if we assigned codepoints to every fictional script and every glyph used in dead scripts, and every variation used in medieval and ancient times, and every future new emoticon or glyph for new scripts or fashionable things. Remember, you need a fairly large number of users for each glyph, otherwise there's no point. We're not going to have per-person glyphs, as no one could understand enough of them for them to be useful. There aren't going to be too many new scripts with users in just the hundreds of thousands, or low single-digits millions. However, if that's wrong, if population gets large enough and regional scripts become a thing... then yeah, we'd run out, but let's hope not -- Unicode is complex enough as it is.
  - WorldMaker 8y ago
    > Remember, you need a fairly large number of users for each glyph, otherwise there's no point. You are forgetting scripts like Linear A and Linear B which Unicode encodes despite only being of use to a few dozen academics or so at a time, and a few dozen (if that) or so (very) historic documents and maybe hundreds of academic papers over time. Popularity/usage isn't a determiner for Unicode encoding of a script. There are plenty of good reasons to encode low usage scripts and glyphs, if they help people communicate when they are used. But yes, I agree that it is probably a long while until the Astral Plane runs out, even encoding every small academically interesting thing along the way. I just don't feel we can discount the possibility that the Astral Plane won't be the last plane simply because we can't imagine today what the "Imaginary Plane" would even contain.
    - cryptonector 8y ago
      I'm not forgetting those or any other historical scripts. I mentioned those. My comment about the number of users of glyphs was in the context of future new glyphs. The supply of old glyphs is pretty limited, so it's easy to accept them all even if the number of users is minute :)
- ubernostrum 8y ago
  I think you may be confused about the meaning of "plane" in Unicode. A plane is a contiguous set of 65,536 (216) code points. The lowest-numbered plane was originally the only one; now it's simply Plane 0 (officially, the "Basic Multilingual Plane"). "Astral Plane" is a collective term to refer to planes beyond that, of which there are now 16, for a total of 17 planes. Unicode currently promises to cap at 17 planes because that's the limit of what UTF-16 can encode with surrogate pairs; anything beyond Plane 16 (officially, "Supplementary Private Use Area-B") would require some other scheme to encode in 16-bit units. Unicode does not use anywhere near the amount of space even those 17 planes represent; you could encode seven full copies of current Unicode in the unassigned and non-reserved space available, and still have room left over (eight full copies if you un-reserve Planes 15 and 16). If we ever truly needed more than 17 planes, we'd have trouble trying to do UTF-16, but that's a problem with UTF-16, not with Unicode. UTF-8 as currently defined would run into trouble beyond 32 planes, but we'd either abandon it for something else, or bolt on some inelegant hack to extend UTF-8 beyond four bytes. So other than maintaining compatibility with existing encodings, there is nothing about Unicode's design that forbids tacking on more planes if and as needed. The hard part, as with so many things in programming, was going from "there's only one" to "there's more than one". If we ever needed enough planes to exhaust both UTF-16 and UTF-8. One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. To put it in perspective, the total set of emoji in Unicode 11.0, which are spread out across multiple blocks in different planes due to historical reasons, add up to less than 2% of the capacity of a single plane, and 0.11% of the available space in a 17-plane Unicode.
  - filmor 8y ago
    The scheme used in UTF8 can encode up to 42 bits with a single start byte, the last start byte being 0xFF followed by 7 bytes of the form 10xxxxxx.
    - ubernostrum 8y ago
      Hence I said "UTF-8 as currently defined". You could produce something that uses the same basic scheme as UTF-8 (using the leading byte to indicate the total number of bytes used for the code point), but it would not be UTF-8 as we know it (which caps at four bytes per code point), and different encoders/decoders would need to be developed.

WorldMaker 8y ago

It's not entirely a jest, but it is funny to think about. The question will be then, as it has been every time before: what do we mean by "Universal" in Unicode? One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. Less of a jest is that there's always more symbols to encode. We've been a wonderfully creative species, and we're still rediscovering old written languages and discovering new bits of physics and math that can benefit from new symbols. We've got tons of interesting fictional language encodings to consider that are currently not likely to be encoded in Unicode today because they are locked in that struggle between the needs of Academia, the interests the Enthusiasts, and the money/complications of Intellectual Property laws. (Klingon, Tolkien's Elvish languages, D'ni, Timelord Sigils, etc.) But if those seesaws tip towards enthusiasts or academia that changes quickly. Meanwhile, creative artists keep inventing new glyphs and languages to encode. That's just our one species on this planet. What if we discover more or get better at encoding existing non-human languages? Then we really get to test that "Universal" bit in the name, huh? (That's also obviously a bit of jest, but an interesting sci-fi hope/yearning there, too. It would be nice to encode some non-human languages for a change.) So, all of the above, based on current assumptions (emojis will slow down, IP law remains vicious, Fermi's Paradox won't been solved in our lifetimes, etc) may still be unlikely to push us past the Astral Plane, but that doesn't make it silly to assume that we may never need to open the next plane.

cryptonector 8y ago
There's always more glyphs to encode, but I think it will be a long time before we reach hundreds of millions of glyphs, even if we assigned codepoints to every fictional script and every glyph used in dead scripts, and every variation used in medieval and ancient times, and every future new emoticon or glyph for new scripts or fashionable things. Remember, you need a fairly large number of users for each glyph, otherwise there's no point. We're not going to have per-person glyphs, as no one could understand enough of them for them to be useful. There aren't going to be too many new scripts with users in just the hundreds of thousands, or low single-digits millions. However, if that's wrong, if population gets large enough and regional scripts become a thing... then yeah, we'd run out, but let's hope not -- Unicode is complex enough as it is.
- WorldMaker 8y ago
  > Remember, you need a fairly large number of users for each glyph, otherwise there's no point. You are forgetting scripts like Linear A and Linear B which Unicode encodes despite only being of use to a few dozen academics or so at a time, and a few dozen (if that) or so (very) historic documents and maybe hundreds of academic papers over time. Popularity/usage isn't a determiner for Unicode encoding of a script. There are plenty of good reasons to encode low usage scripts and glyphs, if they help people communicate when they are used. But yes, I agree that it is probably a long while until the Astral Plane runs out, even encoding every small academically interesting thing along the way. I just don't feel we can discount the possibility that the Astral Plane won't be the last plane simply because we can't imagine today what the "Imaginary Plane" would even contain.
  - cryptonector 8y ago
    I'm not forgetting those or any other historical scripts. I mentioned those. My comment about the number of users of glyphs was in the context of future new glyphs. The supply of old glyphs is pretty limited, so it's easy to accept them all even if the number of users is minute :)
ubernostrum 8y ago
I think you may be confused about the meaning of "plane" in Unicode. A plane is a contiguous set of 65,536 (216) code points. The lowest-numbered plane was originally the only one; now it's simply Plane 0 (officially, the "Basic Multilingual Plane"). "Astral Plane" is a collective term to refer to planes beyond that, of which there are now 16, for a total of 17 planes. Unicode currently promises to cap at 17 planes because that's the limit of what UTF-16 can encode with surrogate pairs; anything beyond Plane 16 (officially, "Supplementary Private Use Area-B") would require some other scheme to encode in 16-bit units. Unicode does not use anywhere near the amount of space even those 17 planes represent; you could encode seven full copies of current Unicode in the unassigned and non-reserved space available, and still have room left over (eight full copies if you un-reserve Planes 15 and 16). If we ever truly needed more than 17 planes, we'd have trouble trying to do UTF-16, but that's a problem with UTF-16, not with Unicode. UTF-8 as currently defined would run into trouble beyond 32 planes, but we'd either abandon it for something else, or bolt on some inelegant hack to extend UTF-8 beyond four bytes. So other than maintaining compatibility with existing encodings, there is nothing about Unicode's design that forbids tacking on more planes if and as needed. The hard part, as with so many things in programming, was going from "there's only one" to "there's more than one". If we ever needed enough planes to exhaust both UTF-16 and UTF-8. One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. To put it in perspective, the total set of emoji in Unicode 11.0, which are spread out across multiple blocks in different planes due to historical reasons, add up to less than 2% of the capacity of a single plane, and 0.11% of the available space in a 17-plane Unicode.
- filmor 8y ago
  The scheme used in UTF8 can encode up to 42 bits with a single start byte, the last start byte being 0xFF followed by 7 bytes of the form 10xxxxxx.
  - ubernostrum 8y ago
    Hence I said "UTF-8 as currently defined". You could produce something that uses the same basic scheme as UTF-8 (using the leading byte to indicate the total number of bytes used for the code point), but it would not be UTF-8 as we know it (which caps at four bytes per code point), and different encoders/decoders would need to be developed.