3 ms·

We can't assume a signed int, as character encodings limit the number of codepoints: "Excluding surrogates and noncharacters leaves 1,111,998 code points availa

by mkl 5y ago

We can't assume a signed int, as character encodings limit the number of codepoints: "Excluding surrogates and noncharacters leaves 1,111,998 code points available for use." -- https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20surrogates%20and%20noncharacters%20leaves%201%2c111%2c998%20code%20points https://en.wikipedia.org/wiki/Unicode#:~:text=Excluding%20su...

thaumasiotes 5y ago
But character encodings don't limit the number of codepoints. Unicode is just a big list of correspondences between an integer and a glyph. There's no limit to how many integers you can assign. Unicode encodings are separate standards that give correspondences between Unicode code points (integers) and byte sequences. If Unicode changes in a way that invalidates an encoding, that just calls for a new encoding.
- mkl 5y ago
  Yes, it could technically be extended, but the transition would be a massive undertaking, so in practice the encodings do limit the number of codepoints. UTF-16, which creates the limitation, is very widely used and required by major programming language standards like ECMAScript. A lot of software still can't cope with codepoints outside the BMP, and they were established with UTF-16 in 1996.
marcosdumay 5y ago
Besides the difference between the abstract and unlimited Unicode and the encodings, our current "modern" encodings, UTF-8 and the new UTF-16 are artificially restricted and can be trivially expanded into a huge number of codepoints just by removing those restrictions.
- mkl 5y ago
  New UTF-16? I'm only aware of the original 1996 one, which uses all of its 20 surrogate-pair bits for the codepoint (unlike UTF-8 which can use bits to extend to more bytes). In my understanding, "just" removing that restriction would mean completely replacing the encoding, like UCS-2 being replaced with UTF-16. The new one may have some overlap, but transitioning to it would still be a huge undertaking, and far from trivial (quite a few programs today still use UCS-2, quarter of a century after UTF-16 was introduced to replace it).
kevin_thibedeau 5y ago
Unicode has been limited to 21-bits for a while so that UTF8 is guaranteed to encode no more than four bytes per code point. It can support the full 32-bit code space but changing now will break a lot of validation code.

by mkl 5y ago

thaumasiotes 5y ago

But character encodings don't limit the number of codepoints. Unicode is just a big list of correspondences between an integer and a glyph. There's no limit to how many integers you can assign. Unicode encodings are separate standards that give correspondences between Unicode code points (integers) and byte sequences. If Unicode changes in a way that invalidates an encoding, that just calls for a new encoding.

mkl 5y ago
Yes, it could technically be extended, but the transition would be a massive undertaking, so in practice the encodings do limit the number of codepoints. UTF-16, which creates the limitation, is very widely used and required by major programming language standards like ECMAScript. A lot of software still can't cope with codepoints outside the BMP, and they were established with UTF-16 in 1996.

marcosdumay 5y ago

Besides the difference between the abstract and unlimited Unicode and the encodings, our current "modern" encodings, UTF-8 and the new UTF-16 are artificially restricted and can be trivially expanded into a huge number of codepoints just by removing those restrictions.

mkl 5y ago
New UTF-16? I'm only aware of the original 1996 one, which uses all of its 20 surrogate-pair bits for the codepoint (unlike UTF-8 which can use bits to extend to more bytes). In my understanding, "just" removing that restriction would mean completely replacing the encoding, like UCS-2 being replaced with UTF-16. The new one may have some overlap, but transitioning to it would still be a huge undertaking, and far from trivial (quite a few programs today still use UCS-2, quarter of a century after UTF-16 was introduced to replace it).

kevin_thibedeau 5y ago

Unicode has been limited to 21-bits for a while so that UTF8 is guaranteed to encode no more than four bytes per code point. It can support the full 32-bit code space but changing now will break a lot of validation code.