13 ms·

My take on this is that it's difficult to communicate the exact details of “Unicode support”, so perhaps it's alright to say that one supports Unicode, generall

My take on this is that it's difficult to communicate the exact details of “Unicode support”, so perhaps it's alright to say that one supports Unicode, generally speaking, but Unicode is complicated and there are corners that aren't supported fully.

cryptonector 6y ago
My take is that it is important to understand the basics: - UTFs - [canonical] equivalence (form, normalization) - [canonical] decomposition - [canonical] precomposition - case mapping - case-folding - localization - the reason for all this complexity (hint: it's not the UC's fault) You don't have to really grok the details, much less memorize them. It does not much more than TFA's ~1,500 words to explain the concepts.
- a1369209993 6y ago
  > - the reason for all this complexity > (hint: it's not the UC's fault) This is a common fallacy, that just because some of the complexity is unavoidable and irreducible, means that the UC hasn't made things much worse than they needed to be with things like code points (which are neither full characters (see eg U+308 combining umlaut) nor single characters (see eg U+1F1 the two letters D and Z)) or emoji. But, yes, with the possible exception of normalization, all of those are things you'd need to deal with in some fashion regardless of Unicode.
  - cryptonector 6y ago
    Even combining codepoints are a thing that predates Unicode and simplifies a number of things, so they were a very useful thing to have in Unicode. First off, diacritical marks have generally been just that: marks that get added to other characters. Notionally that is a great deal like combining codepoints. For example, typewriter fonts, and later ASCII, were designed to have useful overstrike combinations. So you could type á as a<BS>', both in typewriters and in ASCII! Yes, ASCII was a variable-length, multi-byte internationalized encoding for Latin scripts. For example, this is why Spanish used to drop diacritical marks when up-casing: because there was no room in those typewriter fonts for accents on upper-case letters, though nowadays we have much more luxurious fonts, and so you see marks kept on up-cased Spanish a lot more than one used to. Second, diacritical marks as combining marks adds a great deal of flexibility and a fair degree of future-proofing to the system. This is a restatement of the first point, I know. Read on before telling me that flexibility is nice but expensive. Third, it is much simpler to close canonical composition to new compositions by having canonical decomposition than it is to keep adding pre-compositions for every sensible and new character. Part of this is that we have a fairly limited codepoint codespace while pre-compositions are essentially a cartesian explosion, and cartesian explosion means consuming codepoint codespace very quickly. Cartesian explosions complicate software as well (more on this below). This really gets back to the point about flexibility. Fourth, scripts and languages tend to have decomposition strongly built into their rules. It would complicate things to only let them have pre-composed characters! For example, Hangul is really a phonetic script, but is arranged to look syllabic. Now, clearly Hangul is a lot more like Latin in terms of semantics than like Hiragana, say, so you should want to process Hangul text as phonetic (letters) rather than syllabic, but the syllabic aspect of it can't be ignored completely. So Hangul gets precompositions (lots, thanks to cartesian explosion) but even in NFC Hangul is decomposed because that is much more useful for handling in software. Emoji are mostly like ideographic CJK gluphs, except with color. The color aspect is what truly makes emojis a new thing in human scripts, but even color can be a combining mark. Emoji clearly illustrates the fact that our scripts are still evolving. This is a critical thing to understand: our languages and scripts are not set in stone! You could say that there was no need to add emoji all you want, but they were getting added in much software, and users were using them -- eventually the UC was going to have to acknowledge their existence. Aside from emoji, which are truly an innovation (though, again, not the UC's), conceptual decomposition of characters had long been a part of scripts. It was there because it was useful. It's entirely natural, and a very good thing too, that it's there in Unicode as well. Decomposition, and therefore combining codepoints, was probably unavoidable. Regarding digraphs and ligatures, these too are part of our scripts, and we can't very well ignore them completely. This is not to say that the UC didn't make mistakes. Han unification was a mistake that resulted from not inventing UTF-8 sooner. Indeed, the UC did not even invent UTF-8. Some mistakes are going to be unavoidable, but decomposition and combining codepoints are absolutely not a mistake. Finally, we're not likely to ever replace Unicode -- not within the next few decades anyways. However much you might think Unicode is broken, however widespread, right, or wrong that perception might be, Unicode is an unavoidble fact of developers' lives. One might as well accept it as it is, understand it as it is, and move on.
    - a1369209993 6y ago
      > combining codepoints Combining diacritics are a good thing, I'm talking about Unicodes conflation^W deliberate lies that a code point is a meaningful unit of data in it's own right rather than a artifact of using a multi-level encoding from characters like "ä" to bytes like "61 CC 88". > For example, this is why Spanish used to drop diacritical marks when up-casing: because there was no room in those typewriter fonts for accents on upper-case letters Huh, I remembered that, but though it was some badly-designed computer typesetting system. > Third, it is much simpler to close canonical composition to new compositions by having canonical decomposition than it is to keep adding pre-compositions for every sensible and new character. Yeah, this is what conviced me that base+diacritic was a better design than precomposed characters. > Emoji are [...] with color. That - that they are not actually text - is one of several problems with them, yes. > but decomposition and combining codepoints are absolutely not a mistake. Decomposition and combining diacritics are not a mistake. The mistake is treating the decomposed elements as first-class entities. > Finally, we're not likely to ever replace Unicode -- not [soon] anyways. That has been said about literally every bad status quo in the history of problems that existed long enough for people to compain about them. "The dragon is bad!"[0], and no amount of status-quo bias is going to change that. 0: https://www.nickbostrom.com/fable/dragon.html https://www.nickbostrom.com/fable/dragon.html
      - cryptonector 6y ago
        > I'm talking about Unicodes conflation^W deliberate lies that a code point is a meaningful unit of data in it's own right rather than a artifact of using a multi-level encoding from characters What on Earth are you talking about. Where does the UC "lie" like this? Remember, the base codepoint is generally a character in its own right when it's not combined, and combining codepoints are mostly not characters in their own right. > Yeah, this is what conviced me that base+diacritic was a better design than precomposed characters. If anything, precompositions probably exist primarily to make transcoding from other previously-existing codesets (e.g., ISO-8859-*) easier. > > Emoji are [...] with color. > > That - that they are not actually text - is one of several problems with them, yes. How are Kanji text and emoji not? Both are ideographic. What is the distinction? This is not rhetorical -- I'm genuinely curious what you think is the distinction. > > but decomposition and combining codepoints are absolutely not a mistake. > > Decomposition and combining diacritics are not a mistake. The mistake is treating the decomposed elements as first-class entities. The base codepoints are generally characters in their own rights ("first-class entities") while the combining ones are generally not. In what way am I getting that wrong? > > Finally, we're not likely to ever replace Unicode -- not [soon] anyways. > > That has been said about literally every bad status quo in the history of problems that existed long enough for people to compain about them. You are making arguments for why Unicode should be replaced, though I think they are unfounded, but you'll need more than that to get it replaced. Even granting you are right, how would it be made to happen?
- necovek 6y ago
  Well, there's also collation, Unicode collation algorithm, default Unicode collation table, and tailoring it for every locale.

cryptonector 6y ago

My take is that it is important to understand the basics: - UTFs - [canonical] equivalence (form, normalization) - [canonical] decomposition - [canonical] precomposition - case mapping - case-folding - localization - the reason for all this complexity (hint: it's not the UC's fault) You don't have to really grok the details, much less memorize them. It does not much more than TFA's ~1,500 words to explain the concepts.

a1369209993 6y ago
> - the reason for all this complexity > (hint: it's not the UC's fault) This is a common fallacy, that just because some of the complexity is unavoidable and irreducible, means that the UC hasn't made things much worse than they needed to be with things like code points (which are neither full characters (see eg U+308 combining umlaut) nor single characters (see eg U+1F1 the two letters D and Z)) or emoji. But, yes, with the possible exception of normalization, all of those are things you'd need to deal with in some fashion regardless of Unicode.
- cryptonector 6y ago
  Even combining codepoints are a thing that predates Unicode and simplifies a number of things, so they were a very useful thing to have in Unicode. First off, diacritical marks have generally been just that: marks that get added to other characters. Notionally that is a great deal like combining codepoints. For example, typewriter fonts, and later ASCII, were designed to have useful overstrike combinations. So you could type á as a<BS>', both in typewriters and in ASCII! Yes, ASCII was a variable-length, multi-byte internationalized encoding for Latin scripts. For example, this is why Spanish used to drop diacritical marks when up-casing: because there was no room in those typewriter fonts for accents on upper-case letters, though nowadays we have much more luxurious fonts, and so you see marks kept on up-cased Spanish a lot more than one used to. Second, diacritical marks as combining marks adds a great deal of flexibility and a fair degree of future-proofing to the system. This is a restatement of the first point, I know. Read on before telling me that flexibility is nice but expensive. Third, it is much simpler to close canonical composition to new compositions by having canonical decomposition than it is to keep adding pre-compositions for every sensible and new character. Part of this is that we have a fairly limited codepoint codespace while pre-compositions are essentially a cartesian explosion, and cartesian explosion means consuming codepoint codespace very quickly. Cartesian explosions complicate software as well (more on this below). This really gets back to the point about flexibility. Fourth, scripts and languages tend to have decomposition strongly built into their rules. It would complicate things to only let them have pre-composed characters! For example, Hangul is really a phonetic script, but is arranged to look syllabic. Now, clearly Hangul is a lot more like Latin in terms of semantics than like Hiragana, say, so you should want to process Hangul text as phonetic (letters) rather than syllabic, but the syllabic aspect of it can't be ignored completely. So Hangul gets precompositions (lots, thanks to cartesian explosion) but even in NFC Hangul is decomposed because that is much more useful for handling in software. Emoji are mostly like ideographic CJK gluphs, except with color. The color aspect is what truly makes emojis a new thing in human scripts, but even color can be a combining mark. Emoji clearly illustrates the fact that our scripts are still evolving. This is a critical thing to understand: our languages and scripts are not set in stone! You could say that there was no need to add emoji all you want, but they were getting added in much software, and users were using them -- eventually the UC was going to have to acknowledge their existence. Aside from emoji, which are truly an innovation (though, again, not the UC's), conceptual decomposition of characters had long been a part of scripts. It was there because it was useful. It's entirely natural, and a very good thing too, that it's there in Unicode as well. Decomposition, and therefore combining codepoints, was probably unavoidable. Regarding digraphs and ligatures, these too are part of our scripts, and we can't very well ignore them completely. This is not to say that the UC didn't make mistakes. Han unification was a mistake that resulted from not inventing UTF-8 sooner. Indeed, the UC did not even invent UTF-8. Some mistakes are going to be unavoidable, but decomposition and combining codepoints are absolutely not a mistake. Finally, we're not likely to ever replace Unicode -- not within the next few decades anyways. However much you might think Unicode is broken, however widespread, right, or wrong that perception might be, Unicode is an unavoidble fact of developers' lives. One might as well accept it as it is, understand it as it is, and move on.
  - a1369209993 6y ago
    > combining codepoints Combining diacritics are a good thing, I'm talking about Unicodes conflation^W deliberate lies that a code point is a meaningful unit of data in it's own right rather than a artifact of using a multi-level encoding from characters like "ä" to bytes like "61 CC 88". > For example, this is why Spanish used to drop diacritical marks when up-casing: because there was no room in those typewriter fonts for accents on upper-case letters Huh, I remembered that, but though it was some badly-designed computer typesetting system. > Third, it is much simpler to close canonical composition to new compositions by having canonical decomposition than it is to keep adding pre-compositions for every sensible and new character. Yeah, this is what conviced me that base+diacritic was a better design than precomposed characters. > Emoji are [...] with color. That - that they are not actually text - is one of several problems with them, yes. > but decomposition and combining codepoints are absolutely not a mistake. Decomposition and combining diacritics are not a mistake. The mistake is treating the decomposed elements as first-class entities. > Finally, we're not likely to ever replace Unicode -- not [soon] anyways. That has been said about literally every bad status quo in the history of problems that existed long enough for people to compain about them. "The dragon is bad!"[0], and no amount of status-quo bias is going to change that. 0: https://www.nickbostrom.com/fable/dragon.html https://www.nickbostrom.com/fable/dragon.html
    - cryptonector 6y ago
      > I'm talking about Unicodes conflation^W deliberate lies that a code point is a meaningful unit of data in it's own right rather than a artifact of using a multi-level encoding from characters What on Earth are you talking about. Where does the UC "lie" like this? Remember, the base codepoint is generally a character in its own right when it's not combined, and combining codepoints are mostly not characters in their own right. > Yeah, this is what conviced me that base+diacritic was a better design than precomposed characters. If anything, precompositions probably exist primarily to make transcoding from other previously-existing codesets (e.g., ISO-8859-*) easier. > > Emoji are [...] with color. > > That - that they are not actually text - is one of several problems with them, yes. How are Kanji text and emoji not? Both are ideographic. What is the distinction? This is not rhetorical -- I'm genuinely curious what you think is the distinction. > > but decomposition and combining codepoints are absolutely not a mistake. > > Decomposition and combining diacritics are not a mistake. The mistake is treating the decomposed elements as first-class entities. The base codepoints are generally characters in their own rights ("first-class entities") while the combining ones are generally not. In what way am I getting that wrong? > > Finally, we're not likely to ever replace Unicode -- not [soon] anyways. > > That has been said about literally every bad status quo in the history of problems that existed long enough for people to compain about them. You are making arguments for why Unicode should be replaced, though I think they are unfounded, but you'll need more than that to get it replaced. Even granting you are right, how would it be made to happen?
necovek 6y ago
Well, there's also collation, Unicode collation algorithm, default Unicode collation table, and tailoring it for every locale.