4 ms·

How would separate code point break round-tripping specifically?

by zokier 1y ago

sebstefan 1y ago
I don't know about round-tripping of anything but suddenly having an entire nation with keyboard outputting utf-8 on outdated national systems probably designed for Latin1 seems like a tough sell to fix this issue
dhosek 1y ago
The legacy Turkish encoding used ASCII i but a character in the 128–255 range for İ. Remember that not all documents are monolingual so you might have a document with, e.g., both English and Turkish text and in the legacy code page these would use i for both the English and Turkish letter.
- zokier 1y ago
  That didn't answer the question, why would separate code point for Turkish lower-case dotted i break round-tripping?
  - dhosek 1y ago
    Because just because something is in Turkish doesn’t mean it doesn’t also include non-Turkish text. So you end up with weird edge cases when translating mixed text back and forth since it would be a single glyph in legacy Turkish 8-bit text but two glyphs in Unicode so Unicode text that might have “Kırgızistan (English: Kyrgyzstan)” in it under your scheme with a Unicode-Legacy-Unicode roundtrip would encode the i in English as the Turkish dotted i.
    - zokier 1y ago
      If "turkish lower case dotted i" would be separate codepoint, that still wouldn't cause ambiguity like you describe. It would just mean that "U+0069 latin small letter i" would not be (directly) transcodable to the legacy Turkish character set. But that wouldn't really be any different from other similar homoglyph situations, for example "U+0430 cyrillic small letter a" does not transcode to ASCII and that is business as usual. U+0069 not being transcodeable to some legacy encoding is not really a round-tripping problem, vast majority of Unicode codepoints are not transcodable to any single legacy encoding. Round-trip compatibility is really only concern when going from legacy-unicode-legacy; it is naturally expected that not all strings will be able to roundtrip unicode-legacy-unicode.
      - dhosek 1y ago
        EXCEPT that the legacy Cyrillic codepages had separate codepoints for Latin a and Cyrillic а. You’re also making assumptions about the roundtrip preservation that are invalid. The idea is that if a string is encodable in the legacy codepage, you should be able to make the roundtrip. Yes, you can’t roundtrip ⨋ to most legacy codepages, but that’s not the brief.
        zokier 1y ago
        > The idea is that if a string is encodable in the legacy codepage, you should be able to make the roundtrip. But the which strings are encodable in legacy codepage depends on what we define as encodable! If we had separate codepoint for "turkish small letter i" then we could have simply defined that "latin small letter i" is not encodable in legacy turkish codepage, same way that "cyrillic small letter a" is not encodable to turkish legacy codepage. "turkish small letter i" and "latin small letter i" would be just another normal homoglyph pair, same as "cyrillic small letter a" and "latin small letter a".

How would separate code point break round-tripping specifically?

by zokier 1y ago

How would separate code point break round-tripping specifically?

I don't know about round-tripping of anything but suddenly having an entire nation with keyboard outputting utf-8 on outdated national systems probably designed for Latin1 seems like a tough sell to fix this issue

dhosek 1y ago

The legacy Turkish encoding used ASCII i but a character in the 128–255 range for İ. Remember that not all documents are monolingual so you might have a document with, e.g., both English and Turkish text and in the legacy code page these would use i for both the English and Turkish letter.

zokier 1y ago
That didn't answer the question, why would separate code point for Turkish lower-case dotted i break round-tripping?
- dhosek 1y ago
  Because just because something is in Turkish doesn’t mean it doesn’t also include non-Turkish text. So you end up with weird edge cases when translating mixed text back and forth since it would be a single glyph in legacy Turkish 8-bit text but two glyphs in Unicode so Unicode text that might have “Kırgızistan (English: Kyrgyzstan)” in it under your scheme with a Unicode-Legacy-Unicode roundtrip would encode the i in English as the Turkish dotted i.
  - zokier 1y ago
    If "turkish lower case dotted i" would be separate codepoint, that still wouldn't cause ambiguity like you describe. It would just mean that "U+0069 latin small letter i" would not be (directly) transcodable to the legacy Turkish character set. But that wouldn't really be any different from other similar homoglyph situations, for example "U+0430 cyrillic small letter a" does not transcode to ASCII and that is business as usual. U+0069 not being transcodeable to some legacy encoding is not really a round-tripping problem, vast majority of Unicode codepoints are not transcodable to any single legacy encoding. Round-trip compatibility is really only concern when going from legacy-unicode-legacy; it is naturally expected that not all strings will be able to roundtrip unicode-legacy-unicode.
    - dhosek 1y ago
      EXCEPT that the legacy Cyrillic codepages had separate codepoints for Latin a and Cyrillic а. You’re also making assumptions about the roundtrip preservation that are invalid. The idea is that if a string is encodable in the legacy codepage, you should be able to make the roundtrip. Yes, you can’t roundtrip ⨋ to most legacy codepages, but that’s not the brief.
      - zokier 1y ago
        > The idea is that if a string is encodable in the legacy codepage, you should be able to make the roundtrip. But the which strings are encodable in legacy codepage depends on what we define as encodable! If we had separate codepoint for "turkish small letter i" then we could have simply defined that "latin small letter i" is not encodable in legacy turkish codepage, same way that "cyrillic small letter a" is not encodable to turkish legacy codepage. "turkish small letter i" and "latin small letter i" would be just another normal homoglyph pair, same as "cyrillic small letter a" and "latin small letter a".