4 ms·
> Usually, what you want is either the byte or the grapheme cluster. Exactly ! That's what I understood after reading this great post https://tonsky.me/blog/un
by karteum 1y ago
> Usually, what you want is either the byte or the grapheme cluster.
Exactly ! That's what I understood after reading this great post https://tonsky.me/blog/unicode/ https://tonsky.me/blog/unicode/
"Even in the widest encoding, UTF-32, [some grapheme] will still take three 4-byte units to encode. And it still needs to be treated as a single character. If the analogy helps, we can think of the Unicode itself (without any encodings) as being variable-length."
I tend to think it's the biggest design decision in Unicode (but maybe I just don't fully see the need and use-cases beyond emojis. Of course I read the section saying it's used in actual languages, but the few examples described could have been made with a dedicated 32 bits codepoint...)
- panpog 1y agoCan you fit everything into 32 bits? I have no idea, but Hangul and indict scripts seem like they might have a combinatoric explosion of infrequently used characters.
- eviks 1y agoBut they don't have that explosion if you only encode the combinatoric primitives those characters are made of and then use composing rules?
- panpog 1y agoYou still get the combinatoric explosion, but you have more bits to work with. Imagine if you could combine any 9 jamo into a single hangul syllable block. (The real combinatorics is more complicated, and I don't know if it's this bad.) Encoding just the 24 jamo and a a control character requires 25 codepoints. Giving each syllable block its own codepoint would require 24^9>2^32 codepoints.
- eviks 1y ago> Giving each syllable block its own codepoint That's the thing - you wouldn't do that! Only a small subset of frequently used combos would get it's own id, the rest would only be composable