4 ms·

It is surprising to me, because I can mix Czech, English and German just fine on a single screen, even in a single text field. From that perspective having to s

by dvdkon 5y ago

It is surprising to me, because I can mix Czech, English and German just fine on a single screen, even in a single text field. From that perspective having to say "this whole text is in language X" seems backwards.

dmurray 5y ago
OK, yeah. If Unicode hadn't gone with Han unification (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) you'd have the same flexibility in mixing C/J/K.
- rectang 5y ago
  > (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) Representation matters. There should have been plenty of experts who understood that Han unification was problematic. It seems they were not in a position to do anything about it.
  - masklinn 5y ago
    Part of the problem back then is they really really wanted to fit it all in 16 bits, and given there are more hanzi than that to start with this was a bit of an issue (back in the 80s the largest dictionary listed >54000 characters, now it’s >100000, and Japanese and Korean total about 50000 each). A few years later they increased the character space by 5 bits and it wasn’t an issue anymore, but the original legacy of han unification remains.
    - rectang 5y ago
      Yes, and the people who prioritized the 16-bit size were OK with Japanese looking ridiculous in order to achieve their goal. From the article: > if the equivalent symptom was happening with English text, ιҭ wѳuld bє lѳѳκιng sѳmєҭЋιng lικє ҭЋιs. From a page linked by the article, "I Can Text You A Pile of Poo, But I Can’t Write My Name": > To help English readers understand the absurdity of this premise, consider that the Latin alphabet (used by English) and the Cyrillic alphabet (used by Russian) are both derived from Greek. No native English speaker would ever think to try “Greco Unification” and consolidate the English, Russian, German, Swedish, Greek, and other European languages’ alphabets into a single alphabet. If there had been a proposal to sacrifice English in order to cram Unicode into a certain code space size, is there any question that the people on the panel whose first language was English would have quashed it? But people who might have spoken out for Japanese, or for Indian languages, etc. were seemingly not in a position to do anything.
      - jefftk 5y ago
        Greco Unification wouldn't have been a priority because it couldn't have saved enough bits to matter.
        rectang 5y ago
        Yes, of course — but the point is that native English speakers would have blocked such an absurd proposal no matter what. Han Unification is just as absurd and just as unacceptable to a native Japanese speaker as Greco Unification would be to a native English speaker. But Han Unification went through because native Japanese speakers were not in a position to block it. Representation matters.
        verall 5y ago
        Were they? Or were they worried that their characters would be the ones given 4 bytes, and unifying means not squabbling with China and Korea over who goes first? It's not like Japan was not a technical center at the time. Shift-JIS is still around.
        rectang 5y ago
        (EDIT: for screen reader users, a translation follows.) ¢σηѕι∂єяιηg тнє нιѕтσяι¢αℓ αηтιραтну вєтωєєη נαραη αη∂ ¢нιηα ιт'ѕ ρяєρσѕтєяσυѕ тнαт тнє נαραηєѕє ωσυℓ∂ υηιℓαтєяαℓℓу ѕα¢яιƒι¢є тнєιя ¢υℓтυяє ιη ѕυ¢н α ωαу. (EDIT: Considering the historical antipathy between Japan and China it's preposterous that the Japanese would unilaterally sacrifice their culture in such a way.)
    - tdeck 5y ago
      It makes sense given constraints of the time to try to fit the character set into 16 bits, but Unicode has variation selectors, why not use those for the ambiguous characters? They could have easily done something like HAN_CHARACTER_X + VARIANT_KANJI. It would take up an extra 16 bits, but given the density of CJK text relative to Latin text that may not be a big issue. Edit: s/variant/variation/.
      - alpaca128 5y ago
        So such a variation selector would have to be used in front of every individual character? Or could it be applied across a substring, similar to how the text direction is flipped? Either way this seems to be the most painless "fix" for this given the current situation. Setting a language for the whole text fails as soon as the languages are mixed.
      - rectang 5y ago
        Could this be done now?
        deleted 5y ago
        [deleted]
- deleted 5y ago
  [deleted]
jfk13 5y ago
Suppose you're blind and use a screen reader. How does German sound when pronounced by an English text-to-speech engine, or vice versa? Which should be used to read out that "single text field" that contains a mix of languages?
- dvdkon 5y ago
  When reading mixed-language text we can't know if the "Tee" in the middle of an English sentence is German for "tea" or English slang for "t-shirt". We have to use imperfect context clues, so screen readers will have to do the same, it should be doable with today's technology. And if the software doesn't get it right every time, that's fine, because neither do humans.
  - masklinn 5y ago
    > And if the software doesn't get it right every time, that's fine It really is not, a screen reader using the wrong language is way, way worse than a human with the wrong pronunciation. The first time voiceover decided to switch to russian in the middle of english text I thought the os had crashed, the mangling is quite extreme.

by dvdkon 5y ago

dmurray 5y ago

OK, yeah. If Unicode hadn't gone with Han unification (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) you'd have the same flexibility in mixing C/J/K.

rectang 5y ago
> (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) Representation matters. There should have been plenty of experts who understood that Han unification was problematic. It seems they were not in a position to do anything about it.
- masklinn 5y ago
  Part of the problem back then is they really really wanted to fit it all in 16 bits, and given there are more hanzi than that to start with this was a bit of an issue (back in the 80s the largest dictionary listed >54000 characters, now it’s >100000, and Japanese and Korean total about 50000 each). A few years later they increased the character space by 5 bits and it wasn’t an issue anymore, but the original legacy of han unification remains.
  - rectang 5y ago
    Yes, and the people who prioritized the 16-bit size were OK with Japanese looking ridiculous in order to achieve their goal. From the article: > if the equivalent symptom was happening with English text, ιҭ wѳuld bє lѳѳκιng sѳmєҭЋιng lικє ҭЋιs. From a page linked by the article, "I Can Text You A Pile of Poo, But I Can’t Write My Name": > To help English readers understand the absurdity of this premise, consider that the Latin alphabet (used by English) and the Cyrillic alphabet (used by Russian) are both derived from Greek. No native English speaker would ever think to try “Greco Unification” and consolidate the English, Russian, German, Swedish, Greek, and other European languages’ alphabets into a single alphabet. If there had been a proposal to sacrifice English in order to cram Unicode into a certain code space size, is there any question that the people on the panel whose first language was English would have quashed it? But people who might have spoken out for Japanese, or for Indian languages, etc. were seemingly not in a position to do anything.
    - jefftk 5y ago
      Greco Unification wouldn't have been a priority because it couldn't have saved enough bits to matter.
      - rectang 5y ago
        Yes, of course — but the point is that native English speakers would have blocked such an absurd proposal no matter what. Han Unification is just as absurd and just as unacceptable to a native Japanese speaker as Greco Unification would be to a native English speaker. But Han Unification went through because native Japanese speakers were not in a position to block it. Representation matters.
        verall 5y ago
        Were they? Or were they worried that their characters would be the ones given 4 bytes, and unifying means not squabbling with China and Korea over who goes first? It's not like Japan was not a technical center at the time. Shift-JIS is still around.
        rectang 5y ago
        (EDIT: for screen reader users, a translation follows.) ¢σηѕι∂єяιηg тнє нιѕтσяι¢αℓ αηтιραтну вєтωєєη נαραη αη∂ ¢нιηα ιт'ѕ ρяєρσѕтєяσυѕ тнαт тнє נαραηєѕє ωσυℓ∂ υηιℓαтєяαℓℓу ѕα¢яιƒι¢є тнєιя ¢υℓтυяє ιη ѕυ¢н α ωαу. (EDIT: Considering the historical antipathy between Japan and China it's preposterous that the Japanese would unilaterally sacrifice their culture in such a way.)
  - tdeck 5y ago
    It makes sense given constraints of the time to try to fit the character set into 16 bits, but Unicode has variation selectors, why not use those for the ambiguous characters? They could have easily done something like HAN_CHARACTER_X + VARIANT_KANJI. It would take up an extra 16 bits, but given the density of CJK text relative to Latin text that may not be a big issue. Edit: s/variant/variation/.
    - alpaca128 5y ago
      So such a variation selector would have to be used in front of every individual character? Or could it be applied across a substring, similar to how the text direction is flipped? Either way this seems to be the most painless "fix" for this given the current situation. Setting a language for the whole text fails as soon as the languages are mixed.
    - rectang 5y ago
      Could this be done now?
      - deleted 5y ago
        [deleted]
deleted 5y ago
[deleted]

jfk13 5y ago

Suppose you're blind and use a screen reader. How does German sound when pronounced by an English text-to-speech engine, or vice versa? Which should be used to read out that "single text field" that contains a mix of languages?

dvdkon 5y ago
When reading mixed-language text we can't know if the "Tee" in the middle of an English sentence is German for "tea" or English slang for "t-shirt". We have to use imperfect context clues, so screen readers will have to do the same, it should be doable with today's technology. And if the software doesn't get it right every time, that's fine, because neither do humans.
- masklinn 5y ago
  > And if the software doesn't get it right every time, that's fine It really is not, a screen reader using the wrong language is way, way worse than a human with the wrong pronunciation. The first time voiceover decided to switch to russian in the middle of english text I thought the os had crashed, the mangling is quite extreme.