3 ms·

Another aspect of this tragedy is that the creation of UTF-16 has forever limited Unicode to only 17 planes. The original UTF-8 encoding (https://tools.ietf.org

by cesarb 7y ago

Another aspect of this tragedy is that the creation of UTF-16 has forever limited Unicode to only 17 planes. The original UTF-8 encoding (https://tools.ietf.org/html/rfc2044 https://tools.ietf.org/html/rfc2044) used up to six bytes, and could represent a full 31-bit range, which corresponds to 32768 16-bit planes; after UTF-16 became common, UTF-8 was limited to four bytes (https://tools.ietf.org/html/rfc3629 https://tools.ietf.org/html/rfc3629).

kps 7y ago
The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). 1 7 0xxxxxxx 2 11 110xxxxx 10xxxxxx 3 16 1110xxxx 10xxxxxx 10xxxxxx 4 21 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 5 26 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 6 31 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 7 36 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Then, you can either decide to limit the leader to one byte, giving you 42 bits… 8 42 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Or allow multi-byte leaders and continue forever. 9 53 11111111 110xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10 58 11111111 1110xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ⋮ The Intergalactic Confederation will not look kindly on UTF-16.
- cesarb 7y ago
  > Or allow multi-byte leaders and continue forever. That loses one very useful property of UTF-8: you always know, by looking at a single byte, whether it's the first byte of a code point or not. It also loses the property that no valid UTF-8 encoding is a subset of another valid UTF-8 encoding. It's better to stop at 36 bits (which keeps another useful property: you'll never find a 0xFF byte within an UTF-8 string, and you'll only find a 0x00 byte when it's encoding the U+0000 code point).
- cat199 7y ago
  > The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). Naah.. there's UTF-9 and UTF-18 for that.. https://tools.ietf.org/html/rfc4042 https://tools.ietf.org/html/rfc4042
amluto 7y ago
Is this really forever? ISTM the old version could be reintroduced as UTF-8+ without insurmountable problems and with a good degree of compatibility.
userbinator 7y ago
Why would you ever need 31 bits? That's over TWO BILLION distinct codepoints. Only ~137K have been assigned so far, or in other words a little over 17 bits. Given that each additional bit doubles the range, even in the case that characters continue to be assigned at a constant rate, reaching that high is so far into the future that it's not worth thinking about. Unicode is around 30 years old; at the same rate, another 30 years would be needed to get to 18 bits, then another 60, 120, ... On the other hand, it is extremely wasteful to burden all UTF-8 processing code with handling values that will literally never be encountered outside of error cases.
- IshKebab 7y ago
  Well if they keep adding random emoji at the current rate....
  - mark-r 7y ago
    I think that was the point - if they add at the current rate, we'll be OK. If they started adding at an exponential rate we'd have trouble, but that's highly unlikely.

by cesarb 7y ago

kps 7y ago

The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). 1 7 0xxxxxxx 2 11 110xxxxx 10xxxxxx 3 16 1110xxxx 10xxxxxx 10xxxxxx 4 21 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 5 26 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 6 31 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 7 36 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Then, you can either decide to limit the leader to one byte, giving you 42 bits… 8 42 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Or allow multi-byte leaders and continue forever. 9 53 11111111 110xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10 58 11111111 1110xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ⋮ The Intergalactic Confederation will not look kindly on UTF-16.

cesarb 7y ago
> Or allow multi-byte leaders and continue forever. That loses one very useful property of UTF-8: you always know, by looking at a single byte, whether it's the first byte of a code point or not. It also loses the property that no valid UTF-8 encoding is a subset of another valid UTF-8 encoding. It's better to stop at 36 bits (which keeps another useful property: you'll never find a 0xFF byte within an UTF-8 string, and you'll only find a 0x00 byte when it's encoding the U+0000 code point).
cat199 7y ago
> The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). Naah.. there's UTF-9 and UTF-18 for that.. https://tools.ietf.org/html/rfc4042 https://tools.ietf.org/html/rfc4042

amluto 7y ago

Is this really forever? ISTM the old version could be reintroduced as UTF-8+ without insurmountable problems and with a good degree of compatibility.

userbinator 7y ago

Why would you ever need 31 bits? That's over TWO BILLION distinct codepoints. Only ~137K have been assigned so far, or in other words a little over 17 bits. Given that each additional bit doubles the range, even in the case that characters continue to be assigned at a constant rate, reaching that high is so far into the future that it's not worth thinking about. Unicode is around 30 years old; at the same rate, another 30 years would be needed to get to 18 bits, then another 60, 120, ... On the other hand, it is extremely wasteful to burden all UTF-8 processing code with handling values that will literally never be encountered outside of error cases.

IshKebab 7y ago
Well if they keep adding random emoji at the current rate....
- mark-r 7y ago
  I think that was the point - if they add at the current rate, we'll be OK. If they started adding at an exponential rate we'd have trouble, but that's highly unlikely.