4 ms·

20 years later, it should be clear, that utf will stay with us as long as we will read text from screens. So there might be something to the criticism that the

20 years later, it should be clear, that utf will stay with us as long as we will read text from screens. So there might be something to the criticism that the space in UTF will eventually be insufficient.

zaik 4y ago
Is there a limit to how many bytes a code poing can be?
- chrismorgan 4y ago
  Unicode is limited to a little over 20 bits, a bit over a million characters. This limitation is because of UTF-16, which I hold to be the worst thing that ever happened to Unicode. Had it not been for UTF-16, the limit would be 32 bits, four billion characters. As it stands, Unicode is about 25.5% full¹, up from 22.4% a decade ago² and 21% two decades ago³. (These figures are the designated counts; 12.5% of the total space was basically reserved from the start (Unicode 2.0), in control characters, private use areas, surrogates (because UTF-16) and noncharacters.) And by this point, we’re running out of real things to add. (You run out of languages that you want to represent, which leaves you with just the occasional emoji or other symbol that you want to add, and in the last decade emoji have accounted for about 1% of the additions.) Under current policies of what to include, it should easily be good for at least another five thousand years, and I don’t think it’s worth trying to forecast that far in advance. —⁂— ¹ https://www.unicode.org/versions/stats/charcountv14_0.html https://www.unicode.org/versions/stats/charcountv14_0.html ² https://www.unicode.org/versions/stats/charcountv6_1.html https://www.unicode.org/versions/stats/charcountv6_1.html ³ https://www.unicode.org/versions/stats/charcountv3_2.html https://www.unicode.org/versions/stats/charcountv3_2.html
  - ChadNauseam 4y ago
    perhaps when we encounter alien life we’ll have to deprecate UTF-16 and increase the limit
- Someone 4y ago
  Reading https://en.wikipedia.org/wiki/UTF-8#Encoding https://en.wikipedia.org/wiki/UTF-8#Encoding, when we get to 8-byte long UTF-8 code points, the first byte would have to be all ones (11111111) Those would have room for 7 bytes with 6 bits of data each, or 42 bits. Of course, if the need arises (which won’t be soon) we could choose to add another byte to encode the length of the actual byte stream. (Currently, even going beyond four bytes already would break code, as Unicode explicitly limited the UTF-8 encoding in https://datatracker.ietf.org/doc/html/rfc3629 https://datatracker.ietf.org/doc/html/rfc3629 to allow round-tripping to UTF-16. I also think the above goes beyond what was proposed historically (https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History))

zaik 4y ago

Is there a limit to how many bytes a code poing can be?

chrismorgan 4y ago
Unicode is limited to a little over 20 bits, a bit over a million characters. This limitation is because of UTF-16, which I hold to be the worst thing that ever happened to Unicode. Had it not been for UTF-16, the limit would be 32 bits, four billion characters. As it stands, Unicode is about 25.5% full¹, up from 22.4% a decade ago² and 21% two decades ago³. (These figures are the designated counts; 12.5% of the total space was basically reserved from the start (Unicode 2.0), in control characters, private use areas, surrogates (because UTF-16) and noncharacters.) And by this point, we’re running out of real things to add. (You run out of languages that you want to represent, which leaves you with just the occasional emoji or other symbol that you want to add, and in the last decade emoji have accounted for about 1% of the additions.) Under current policies of what to include, it should easily be good for at least another five thousand years, and I don’t think it’s worth trying to forecast that far in advance. —⁂— ¹ https://www.unicode.org/versions/stats/charcountv14_0.html https://www.unicode.org/versions/stats/charcountv14_0.html ² https://www.unicode.org/versions/stats/charcountv6_1.html https://www.unicode.org/versions/stats/charcountv6_1.html ³ https://www.unicode.org/versions/stats/charcountv3_2.html https://www.unicode.org/versions/stats/charcountv3_2.html
- ChadNauseam 4y ago
  perhaps when we encounter alien life we’ll have to deprecate UTF-16 and increase the limit
Someone 4y ago
Reading https://en.wikipedia.org/wiki/UTF-8#Encoding https://en.wikipedia.org/wiki/UTF-8#Encoding, when we get to 8-byte long UTF-8 code points, the first byte would have to be all ones (11111111) Those would have room for 7 bytes with 6 bits of data each, or 42 bits. Of course, if the need arises (which won’t be soon) we could choose to add another byte to encode the length of the actual byte stream. (Currently, even going beyond four bytes already would break code, as Unicode explicitly limited the UTF-8 encoding in https://datatracker.ietf.org/doc/html/rfc3629 https://datatracker.ietf.org/doc/html/rfc3629 to allow round-tripping to UTF-16. I also think the above goes beyond what was proposed historically (https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History))