3 ms·

1) there's only ~150k unicode values defined. If we assume a signed int for available space, we have 2,147,333,647 of 2,147,483,647 remaining. moreso if the int

by knome 5y ago

1) there's only ~150k unicode values defined. If we assume a signed int for available space, we have 2,147,333,647 of 2,147,483,647 remaining. moreso if the int is unsigned. We're fine. 2) they use values that combine like ligatures to create the variants of values. there isn't a combinatorial explosion because color is a modifier value, and sex, and then the underlying symbol. It's not a unique symbol for each combination. IPv4 ran down because everything needs an IP to be on the net and there are more humans than available addresses, and more gear than humans. We don't need different characters per human, only to document existing languages and to account for the slow growth of modern hieroglyphs.

monsieurbanana 5y ago
> We don't need different characters per human Unicode NFTs here we come
cygx 5y ago
If we assume a signed int for available space Note that as it is currently defined, the Unicode codespace ranges from U+0000 to U+10FFFF, with some reserved codepoints (eg to encode surrogate pairs), yielding a total number of 1,112,064 assignable code points.
- throw0101a 5y ago
  1,112,064 code points ought to be enough for anybody. — Bill Gates
- chrismorgan 5y ago
  > as it is currently defined I find it completely implausible that this will ever change: the current size is baked in too heavily. • The abomination UTF-16, which is distressingly popular, cannot possibly support it. Replacing UTF-16 would be a massive upheaval in many ecosystems (e.g. JavaScript, Qt, Windows), and there’s no real prospect of most of those environments moving away from UTF-16, because it’s a massive breaking change for them by now. Rather, if the code space were running out, they’d devise something along the lines of second-level surrogate pairs. (And then we’d curse UTF-16 even more, because it’d have ruined Unicode for everyone again.) • All code that performs Unicode validation (which isn’t as much as it should be, but is still probably a majority) would need to be upgraded. Any systems not upgraded would either mangle or more commonly fail on new characters. • UTF-8 software would also need to be adjusted, since it’s artificially limited to the 21-bit space; and it wouldn’t be just a matter of flipping a few switches here and there to remove that limit—there will be lots of small places that bake in the the assumption that representing a scalar value requires no more than four UTF-8 code units.
  - nukemaster 5y ago
mkl 5y ago
We can't assume a signed int, as character encodings limit the number of codepoints: "Excluding surrogates and noncharacters leaves 1,111,998 code points available for use." -- https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20surrogates%20and%20noncharacters%20leaves%201%2c111%2c998%20code%20points https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20su...
- thaumasiotes 5y ago
  But character encodings don't limit the number of codepoints. Unicode is just a big list of correspondences between an integer and a glyph. There's no limit to how many integers you can assign. Unicode encodings are separate standards that give correspondences between Unicode code points (integers) and byte sequences. If Unicode changes in a way that invalidates an encoding, that just calls for a new encoding.
  - mkl 5y ago
    Yes, it could technically be extended, but the transition would be a massive undertaking, so in practice the encodings do limit the number of codepoints. UTF-16, which creates the limitation, is very widely used and required by major programming language standards like ECMAScript. A lot of software still can't cope with codepoints outside the BMP, and they were established with UTF-16 in 1996.
- marcosdumay 5y ago
  Besides the difference between the abstract and unlimited Unicode and the encodings, our current "modern" encodings, UTF-8 and the new UTF-16 are artificially restricted and can be trivially expanded into a huge number of codepoints just by removing those restrictions.
  - mkl 5y ago
    New UTF-16? I'm only aware of the original 1996 one, which uses all of its 20 surrogate-pair bits for the codepoint (unlike UTF-8 which can use bits to extend to more bytes). In my understanding, "just" removing that restriction would mean completely replacing the encoding, like UCS-2 being replaced with UTF-16. The new one may have some overlap, but transitioning to it would still be a huge undertaking, and far from trivial (quite a few programs today still use UCS-2, quarter of a century after UTF-16 was introduced to replace it).
masklinn 5y ago
> If we assume a signed int for available space While UTF8 was originally defined as able to encode 31 bits, because of the limitations of UTF-16 RFC 3629 explicitly restricted the unicode code-space to 21 bits (or about 1.1 million codepoints).

by knome 5y ago

monsieurbanana 5y ago

> We don't need different characters per human Unicode NFTs here we come

cygx 5y ago

If we assume a signed int for available space Note that as it is currently defined, the Unicode codespace ranges from U+0000 to U+10FFFF, with some reserved codepoints (eg to encode surrogate pairs), yielding a total number of 1,112,064 assignable code points.

throw0101a 5y ago
1,112,064 code points ought to be enough for anybody. — Bill Gates
chrismorgan 5y ago
> as it is currently defined I find it completely implausible that this will ever change: the current size is baked in too heavily. • The abomination UTF-16, which is distressingly popular, cannot possibly support it. Replacing UTF-16 would be a massive upheaval in many ecosystems (e.g. JavaScript, Qt, Windows), and there’s no real prospect of most of those environments moving away from UTF-16, because it’s a massive breaking change for them by now. Rather, if the code space were running out, they’d devise something along the lines of second-level surrogate pairs. (And then we’d curse UTF-16 even more, because it’d have ruined Unicode for everyone again.) • All code that performs Unicode validation (which isn’t as much as it should be, but is still probably a majority) would need to be upgraded. Any systems not upgraded would either mangle or more commonly fail on new characters. • UTF-8 software would also need to be adjusted, since it’s artificially limited to the 21-bit space; and it wouldn’t be just a matter of flipping a few switches here and there to remove that limit—there will be lots of small places that bake in the the assumption that representing a scalar value requires no more than four UTF-8 code units.
- nukemaster 5y ago

mkl 5y ago

We can't assume a signed int, as character encodings limit the number of codepoints: "Excluding surrogates and noncharacters leaves 1,111,998 code points available for use." -- https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20surrogates%20and%20noncharacters%20leaves%201%2c111%2c998%20code%20points https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20su...

thaumasiotes 5y ago
But character encodings don't limit the number of codepoints. Unicode is just a big list of correspondences between an integer and a glyph. There's no limit to how many integers you can assign. Unicode encodings are separate standards that give correspondences between Unicode code points (integers) and byte sequences. If Unicode changes in a way that invalidates an encoding, that just calls for a new encoding.
- mkl 5y ago
  Yes, it could technically be extended, but the transition would be a massive undertaking, so in practice the encodings do limit the number of codepoints. UTF-16, which creates the limitation, is very widely used and required by major programming language standards like ECMAScript. A lot of software still can't cope with codepoints outside the BMP, and they were established with UTF-16 in 1996.
marcosdumay 5y ago
Besides the difference between the abstract and unlimited Unicode and the encodings, our current "modern" encodings, UTF-8 and the new UTF-16 are artificially restricted and can be trivially expanded into a huge number of codepoints just by removing those restrictions.
- mkl 5y ago
  New UTF-16? I'm only aware of the original 1996 one, which uses all of its 20 surrogate-pair bits for the codepoint (unlike UTF-8 which can use bits to extend to more bytes). In my understanding, "just" removing that restriction would mean completely replacing the encoding, like UCS-2 being replaced with UTF-16. The new one may have some overlap, but transitioning to it would still be a huge undertaking, and far from trivial (quite a few programs today still use UCS-2, quarter of a century after UTF-16 was introduced to replace it).

masklinn 5y ago

> If we assume a signed int for available space While UTF8 was originally defined as able to encode 31 bits, because of the limitations of UTF-16 RFC 3629 explicitly restricted the unicode code-space to 21 bits (or about 1.1 million codepoints).