4 ms·

> Unicode has problems where the same thing is represented several different ways. That's needed for compatibility with legacy encodings.

derefr 3y ago
Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)
- avgcorrection 3y ago
  Good points.
- farnsworth 3y ago
  I hadn't known that point about sort order. Wait, so suppose a language that uses the same alphabet as English used a reversed alphabet ordering. Would unicode include z-a somewhere else to support that language?
  - derefr 3y ago
    Probably, yeah. If it's got a fundamentally-different collation, then it's not the same script as the Latin alphabet per se. However, due to how annoying it is to deal with languages with the same glyphs but a different order ("I don't know where to find things in this translation dictionary!"), languages that share a set of glyphs have tended — ever since democratized access to printing, and dictionaries to create "canonical" orderings — to gravitate toward a shared ordering for the common-denominator subsets of their alphabets. German, for example, has an alphabet that's like the Latin alphabet but with some "extra" letters — but, however they did it way back when, the German alphabet today "embeds" the Latin alphabet in Latin order at the beginning, and then puts all the extra letters at the end. So German doesn't need another set of Unicode code-points for A-Z; it just needs codepoints for those extra letters.
    - carlmr 3y ago
      >So German doesn't need another set of Unicode code-points for A-Z; it just needs codepoints for those extra letters. The extra letters would sort differently though right? I wouldn't expect ö/ä/ü to sort after z. For ß I guess it's not a concern since it never appears at the start of words.
  - yorwba 3y ago
    No, they would add new collation rules to the Common Locale Data Repository, similar to the ones that define that Y sorts between I and J in Lithuanian: https://www.unicode.org/cldr/charts/42/collation/lt.html https://www.unicode.org/cldr/charts/42/collation/lt.html
  - thaumasiotes 3y ago
    No, that's not a relevant concern. In the same way, Unicode sees no problem with assigning LATIN CAPITAL LETTER I (U+0049) and LATIN SMALL LETTER I (U+0069) to the glyphs of the Turkish alphabet that have those appearances, despite the fact that in Turkish they are two different letters. (And therefore capitalization has to be done by an algorithm that is aware of what language you want to do capitalization in.) By contrast, GREEK CAPITAL LETTER A is U+0391, separate from LATIN CAPITAL LETTER A. There is no principled reason for this.
- nradov 3y ago
  Now I'm curious about how many mathematicians have published papers using the wrong pi code point. Do journal editors check for that sort of thing?
  - drbaba 3y ago
    Mathematicians usually write papers in LaTeX, which predates Unicode and doesn’t rely on it. You then just type \pi in a math context and get the right symbol. (Though here are modern LaTeX implementations that support Unicode input as well.)

derefr 3y ago

Canonically-equivalent codepoints, yes, exist for embeddings of legacy encodings into Unicode. But when most people — including the GGP poster — complain about this, they aren't usually thinking of canonically-equivalent codepoints, but rather of homoglyphs: codepoints that just happen to have the same conventional pictorial representation. But which have different (collation/splitting/etc) properties; or, more interestingly, different (machine-readable!) semantics. Most often, phishing "text confusion" is done using homoglyphs. (And, in fact, in some text-processing libraries — e.g. Punycode — "text confusion" can only happen using homoglyphs, as canonically-equivalent Unicode codepoints get normalized to just one option.) The reason that the homoglyphic Latin "M" and Cyrillic "М" codepoints exist in Unicode isn't for legacy reasons. It's rather because Cyrillic "M" sorts after Cyrillic "Р" (and both of them come before Cyrillic "С"!) You can't have alternate collation orderings if you're using the same codepoints for both languages. The reason that both Greek "π" and mathematical "𝜋" exist in Unicode, is because one codepoint "is" a letter, and the other codepoint "is" a symbol. This "is"-ness has nothing to do with the glyphs, but rather is basically a type system inside Unicode, that user-agents besides those that render text often rely upon. "𝜋" (the symbol) can carry a particular semantic meaning in text to machine user-agents, that "π" (the letter) does not. Translation systems should treat them differently. Dictionaries should treat them differently. Filter rules for usernames should treat them differently. Etc. (And that's beside the fact that they also have different Unicode properties. "π", as a letter, doesn't create implicit split-boundaries on either side of it; while "𝜋", as a symbol, does.)

avgcorrection 3y ago
Good points.
farnsworth 3y ago
I hadn't known that point about sort order. Wait, so suppose a language that uses the same alphabet as English used a reversed alphabet ordering. Would unicode include z-a somewhere else to support that language?
- derefr 3y ago
  Probably, yeah. If it's got a fundamentally-different collation, then it's not the same script as the Latin alphabet per se. However, due to how annoying it is to deal with languages with the same glyphs but a different order ("I don't know where to find things in this translation dictionary!"), languages that share a set of glyphs have tended — ever since democratized access to printing, and dictionaries to create "canonical" orderings — to gravitate toward a shared ordering for the common-denominator subsets of their alphabets. German, for example, has an alphabet that's like the Latin alphabet but with some "extra" letters — but, however they did it way back when, the German alphabet today "embeds" the Latin alphabet in Latin order at the beginning, and then puts all the extra letters at the end. So German doesn't need another set of Unicode code-points for A-Z; it just needs codepoints for those extra letters.
  - carlmr 3y ago
    >So German doesn't need another set of Unicode code-points for A-Z; it just needs codepoints for those extra letters. The extra letters would sort differently though right? I wouldn't expect ö/ä/ü to sort after z. For ß I guess it's not a concern since it never appears at the start of words.
- yorwba 3y ago
  No, they would add new collation rules to the Common Locale Data Repository, similar to the ones that define that Y sorts between I and J in Lithuanian: https://www.unicode.org/cldr/charts/42/collation/lt.html https://www.unicode.org/cldr/charts/42/collation/lt.html
- thaumasiotes 3y ago
  No, that's not a relevant concern. In the same way, Unicode sees no problem with assigning LATIN CAPITAL LETTER I (U+0049) and LATIN SMALL LETTER I (U+0069) to the glyphs of the Turkish alphabet that have those appearances, despite the fact that in Turkish they are two different letters. (And therefore capitalization has to be done by an algorithm that is aware of what language you want to do capitalization in.) By contrast, GREEK CAPITAL LETTER A is U+0391, separate from LATIN CAPITAL LETTER A. There is no principled reason for this.
nradov 3y ago
Now I'm curious about how many mathematicians have published papers using the wrong pi code point. Do journal editors check for that sort of thing?
- drbaba 3y ago
  Mathematicians usually write papers in LaTeX, which predates Unicode and doesn’t rely on it. You then just type \pi in a math context and get the right symbol. (Though here are modern LaTeX implementations that support Unicode input as well.)