3 ms·

Tangentially, I always found it strange that five and six byte utf-8 sequences are illegal. My naive thought is that it provided a good amount of future proofin

by somat 4y ago

Tangentially, I always found it strange that five and six byte utf-8 sequences are illegal. My naive thought is that it provided a good amount of future proofing for when Unicode needs to expand again.

dhosek 4y ago
Unicode was originally conceived of as a 16-bit encoding. Han unification was necessary to squeeze everything into 16 bits, but it (kind of) worked. Going from 16 bits to 20 bits¹ might not seem like a big increase, but remember that this is an exponential progression, and not a linear one. Those four bits increase the capacity of Unicode by a factor of 16. The amount of empty space is much greater than we’re likely to ever use. Furthermore, there are costs (albeit not necessarily large ones) to having those empty code points—even with efficient use of space using 2-level lookup tables, increasing the capacity of Unicode beyond its current size would add time and space complexity to code for doing things like identifying character codes, grapheme boundaries, case folding, sorting, etc. ⸻ 1. Technically, its 21 bits, but since the maximum value for a code point is 0x10ffff, we can round down to 0xfffff—which means we’re treating the original 16-bit capacity of Unicode as a rounding error!
toast0 4y ago
five and six byte sequences are defined as illegal, because they encode code points that don't fit in utf-16. Of the 1,112,064 assignable code points allowed by UTF-16, we've only assigned 149,186 in the past 30 years, so we've probably got a couple centuries before we'll run out. Deal with it then.
- bentley 4y ago
  I’m surprised that you were downvoted, because you’re totally right. UTF‐16 can represent almost a 21‐bit range—yes, greater than 16—thanks to surrogate pairs. The largest codepoint representable in UTF‐16 is U+10FFFF. That’s why UTF‐8 dropped the 5‐ and 6‐byte sequences, why even UTF‐32 is limited to 21 bits, and why U+D800 to U+DFFF (the range UTF‐16 uses to encode surrogate pairs) are explicitly disallowed from being represented in UTF‐8 and UTF‐32.

by somat 4y ago

dhosek 4y ago

Unicode was originally conceived of as a 16-bit encoding. Han unification was necessary to squeeze everything into 16 bits, but it (kind of) worked. Going from 16 bits to 20 bits¹ might not seem like a big increase, but remember that this is an exponential progression, and not a linear one. Those four bits increase the capacity of Unicode by a factor of 16. The amount of empty space is much greater than we’re likely to ever use. Furthermore, there are costs (albeit not necessarily large ones) to having those empty code points—even with efficient use of space using 2-level lookup tables, increasing the capacity of Unicode beyond its current size would add time and space complexity to code for doing things like identifying character codes, grapheme boundaries, case folding, sorting, etc. ⸻ 1. Technically, its 21 bits, but since the maximum value for a code point is 0x10ffff, we can round down to 0xfffff—which means we’re treating the original 16-bit capacity of Unicode as a rounding error!

toast0 4y ago

five and six byte sequences are defined as illegal, because they encode code points that don't fit in utf-16. Of the 1,112,064 assignable code points allowed by UTF-16, we've only assigned 149,186 in the past 30 years, so we've probably got a couple centuries before we'll run out. Deal with it then.

bentley 4y ago
I’m surprised that you were downvoted, because you’re totally right. UTF‐16 can represent almost a 21‐bit range—yes, greater than 16—thanks to surrogate pairs. The largest codepoint representable in UTF‐16 is U+10FFFF. That’s why UTF‐8 dropped the 5‐ and 6‐byte sequences, why even UTF‐32 is limited to 21 bits, and why U+D800 to U+DFFF (the range UTF‐16 uses to encode surrogate pairs) are explicitly disallowed from being represented in UTF‐8 and UTF‐32.