5 ms·

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use th

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

rurban 2y ago
No, the real solution is to follow the unicode security guidelines for identifiers. Esp. on linux, where the silly garbage-in, garbage-out mantra doesn't fly with identifiers, because identifiers need to stay identifiable. Apple HPFS did some things right. They did at least NFD. But linux insanities brought them back to -Whomoglyph attacks
wheybags 2y ago
I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing becomes super easy like in the old ascii only k&r c example code. I'm not 100% certain, but I don't think there's a widely used language that couldn't be represented by that. Of course, you lose round trip ability with legacy encodings, which is why we have the mess that is unicode. Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Oh well, it's too late now so I guess we just accept it.
- jerf 2y ago
  For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly. I don't think we're there yet. I think if someone did make a complete list of "valid" emoji right now, which for the sake of argument I'll call "formally defined in the Unicode standard", it would even on an absolute scale look like we're a long ways away from a full 32-bits of valid combinations. But you have to think of this on the log scale because this is about "bits" and those four-person families are already quite a long ways along to a full 32 bits. It wouldn't take much more customization, or the formal addition of more people in a group, to get there. And someone who knows more about Unicode than I do may be able to establish that there are already in the standard ways to get to more than 32 bits' worth of data in a single standardized glyph; I certainly wouldn't bet much against that already being true. (Personally, I'll go with "worse". In hindsight, we should probably have frozen Unicode into the original Docomo (and the other phone company that had them) emoji necessary for interoperability, and then created the emoji as an extension into Unicode. It seems like it would be useful to "support Unicode" without having to come with the complete understanding of what is increasingly the most complicated "language" in Unicode; forget doing good Arabic rendering or trying to understand an ideographic language, the emojis blow all that complexity away now. But here we are.) [1]: https://unicode.org/emoji/charts/emoji-zwj-sequences.html https://unicode.org/emoji/charts/emoji-zwj-sequences.html [2]: https://www.unicode.org/reports/tr51/#Multi_Person_Groupings https://www.unicode.org/reports/tr51/#Multi_Person_Groupings
  - jerf 2y ago
    I forgot about "Zalgo". You can definitely get more than 4 billion glyphs, indeed a great deal more, by stacking on modifiers. I can see an argument that that's not really a "valid" use case that we need to worry about too much, though. Emoticons are well on their way to having more possible fully legal, fully intended outputs that go beyond what 32 bits could specify.
- extraduder_ire 2y ago
  >Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Why not? They're just two (or more) characters from a special set next to each other that a font may combine. (and some ad-hoc ZWJ sequences) I don't think windows even ships a font that does that by default.
- ianburrell 2y ago
  Grapheme clusters are locale dependent. Also, if you aren't allowing combining characters, then you are going to need lots of extra codepoints. In some languages, like Indian ones, vowels are combining characters. Or there are languages where multiple code points produce grapheme cluster, like Hangul syllables. You are going to need a lot more code points to represent all possible strings. Text processing is going to be much harder cause there a thousand different representations of Hangul character. Also, backspace is locale dependent. In some languages, backspace removes the accent, which makes sense with combining characters, and other it removes the whole character. Which is going to be fun when whole syllable is code point. Languages are hard, there is no way to make them simple.

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

rurban 2y ago

No, the real solution is to follow the unicode security guidelines for identifiers. Esp. on linux, where the silly garbage-in, garbage-out mantra doesn't fly with identifiers, because identifiers need to stay identifiable. Apple HPFS did some things right. They did at least NFD. But linux insanities brought them back to -Whomoglyph attacks

wheybags 2y ago

I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing becomes super easy like in the old ascii only k&r c example code. I'm not 100% certain, but I don't think there's a widely used language that couldn't be represented by that. Of course, you lose round trip ability with legacy encodings, which is why we have the mess that is unicode. Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Oh well, it's too late now so I guess we just accept it.

jerf 2y ago
For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly. I don't think we're there yet. I think if someone did make a complete list of "valid" emoji right now, which for the sake of argument I'll call "formally defined in the Unicode standard", it would even on an absolute scale look like we're a long ways away from a full 32-bits of valid combinations. But you have to think of this on the log scale because this is about "bits" and those four-person families are already quite a long ways along to a full 32 bits. It wouldn't take much more customization, or the formal addition of more people in a group, to get there. And someone who knows more about Unicode than I do may be able to establish that there are already in the standard ways to get to more than 32 bits' worth of data in a single standardized glyph; I certainly wouldn't bet much against that already being true. (Personally, I'll go with "worse". In hindsight, we should probably have frozen Unicode into the original Docomo (and the other phone company that had them) emoji necessary for interoperability, and then created the emoji as an extension into Unicode. It seems like it would be useful to "support Unicode" without having to come with the complete understanding of what is increasingly the most complicated "language" in Unicode; forget doing good Arabic rendering or trying to understand an ideographic language, the emojis blow all that complexity away now. But here we are.) [1]: https://unicode.org/emoji/charts/emoji-zwj-sequences.html https://unicode.org/emoji/charts/emoji-zwj-sequences.html [2]: https://www.unicode.org/reports/tr51/#Multi_Person_Groupings https://www.unicode.org/reports/tr51/#Multi_Person_Groupings
- jerf 2y ago
  I forgot about "Zalgo". You can definitely get more than 4 billion glyphs, indeed a great deal more, by stacking on modifiers. I can see an argument that that's not really a "valid" use case that we need to worry about too much, though. Emoticons are well on their way to having more possible fully legal, fully intended outputs that go beyond what 32 bits could specify.
extraduder_ire 2y ago
>Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Why not? They're just two (or more) characters from a special set next to each other that a font may combine. (and some ad-hoc ZWJ sequences) I don't think windows even ships a font that does that by default.
ianburrell 2y ago
Grapheme clusters are locale dependent. Also, if you aren't allowing combining characters, then you are going to need lots of extra codepoints. In some languages, like Indian ones, vowels are combining characters. Or there are languages where multiple code points produce grapheme cluster, like Hangul syllables. You are going to need a lot more code points to represent all possible strings. Text processing is going to be much harder cause there a thousand different representations of Hangul character. Also, backspace is locale dependent. In some languages, backspace removes the accent, which makes sense with combining characters, and other it removes the whole character. Which is going to be fun when whole syllable is code point. Languages are hard, there is no way to make them simple.