5 ms·
> I find it confusing when Unicode "shadows" of normal letters exist, and those are of course also dangerous in some cases when they can be mis-interpreted for
by jjmarr 11mo ago
> I find it confusing when Unicode "shadows" of normal letters exist, and those are of course also dangerous in some cases when they can be mis-interpreted for the letter they look more or less exactly like
Isn't this why Unicode normalization exists? This would let you compare Unicode letters and determine if they are canonically equivalent.
- ComputerGuru 11mo agoNormalization wouldn’t address this.
- happytoexplain 11mo agoWhat do you mean? All four normal forms of the Kelvin 'K' are the Latin 'K', as far as I can tell.
- nwellnhof 11mo agoNormalization forms NFKC and NFKD that also handle compatibility equivalence do.
- mananaysiempre 11mo agoA few deprecated characters, including the Kelvin and Ångström symbols, are in fact canonically equivalent to their replacements and not just compatibility equivalent, so plain NFC/NFD is enough. (It’s generally better to avoid NFKC/NFKD normalizations unless you fully understand the implications, as they do lose meaning and at the same time do not account for all possible confusables.)
- Sesse__ 11mo agoIt's why the Unicode Collation Algorithm exists. If you look in allkeys.txt (the base UCA data, used if you don't have language-specific stuff in your comparisons) for the two code points in question, you'll find: 004B ; [.2514.0020.0008] # LATIN CAPITAL LETTER K 212A ; [.2514.0020.0008] # KELVIN SIGN The numbers in the brackets are values on level 1 (base), level 2 (typically used for accents), level 3 (typically used for case). So they are to compare identical under the UCA, in almost every case except for if you really need a tiebreaker. Compare e.g. : 1D424 ; [.2514.0020.0005] # MATHEMATICAL BOLD SMALL K which would compare equal to those under a case-insensitive accent-sensitive collation, but _not_a case-sensitive one (case-sensitive collations are always accent-sensitive, too).
- happytoexplain 11mo agoAre the meanings for the levels for each code point defined somewhere (accent, casing, etc)?
- Sesse__ 11mo agoTypically it is defined by the collation. For the default collation, where all the weights are as in the file, it's none/accent/accent+case. But if you go to e.g. Japanese, you can have a fourth level of “kana-sensitive” (which distinguishes between e.g. katakana and hiragana).