10 ms·

That's extremely interesting, if not depressing. So, if I have to display user-entered text (usernames, posts, comments, messages, form data, etc), and I want

That's extremely interesting, if not depressing. So, if I have to display user-entered text (usernames, posts, comments, messages, form data, etc), and I want to do The Right Thing™: - I cannot rely on user locale, because it might be set to something generic like English, or the user may be bi-lingual. - I cannot rely on location, because the user may be traveling to a different CJK region, or somewhere else altogether. - I cannot set a single lang: attribute for the whole page because it'll be wrong for the other two languages. - The string alone is not sufficient to identify the language because you can write valid sentences in different CJK languages with the same codepoints. - I cannot have a per-user language setting, because users may be bi-lingual. What does that leave me? A dropdown list "C/J/K/Other" besides every single text field? I'm chucking this on my pile of examples of software development being hopelessly broken by design, along with "unix time is non-monotonic and discontinuous at random" (hint: what's the unix time exactly 1e8 seconds, ~3 years, from now? Answer: it's up to the astronomers[1]!). [1]: https://en.wikipedia.org/wiki/Unix_time#Leap_seconds https://en.wikipedia.org/wiki/Unix_time#Leap_seconds Edit: actually, even the dropdown list is insufficient because it only allows one language per string! How is a Japanese user asking for help learning Chinese supposed to write?

zokier 5y ago
> unix time is non-monotonic and discontinuous at random Well, depends on your definition of unix time; if you use time() as the definition then it is actually monotonic because the integral part only repeats on leap seconds?
dathinab 5y ago
The problem is btw. not specific to Han unification/asian languages. It's that for every western language if you use screen readers.
- the_other 5y ago
  My thought whilst reading the article was that the Han unification would actually help screen readers. IIUC The meaning of the glyph is the same across all the languages, so the screen reader will get the correct meaning and can present it according to local settings. The problem with the European languages is that the different characters (letter variations, accent variations) can change the meaning of the word they're part of. Or have I misunderstood?
  - lifthrasiir 5y ago
    The same Han character can have wildly different pronunciations even in a single language (being a logogram, they represent a word and not a sound). KS X 1001, the primary Korean character set, even duplicated same characters according to their readings so that they can be almost [1] correctly converted back to Hangul. In practice they didn't work well though, and Unicode assigns all but one duplicate characters into the compatibility region. [1] These readings didn't take account for systematic variations like the initial sound law (두음법칙, for example 이 vs. 리 at the beginning of words).
  - soraminazuki 5y ago
    You can't split CJK sentences into individual letters, inspect them one by one and decipher its exact meaning. If you present Chinese writing to a Japanese speaker, they'd only see complete gibberish consisting of letters they may or may not recognize. It also works the other way around too. On top of that, Kanjis aren't the only type of letters Japanese folks use. They also use Hiraganas and Katakanas, which are phonetic symbols and totally unrecognizable to non-Japanese speakers.
- Snild 5y ago
  I'm not sure I understand what you're saying here. Are you saying that screen readers don't know which language to interpret text as?
  - dathinab 5y ago
    If you don't tell them, yes. As many websites do not convey the language properly they try to fix that using some heuristics, but especially for small texts (like messages) that can easily fail hard leading to completely wrong pronunciation. Mail for example tried to fix that by allowing you to annotate the language for UTF-8 embedded in mail headers and the content. Ironically while that mechanism works for display names it doesn't work for email addresses itself. And I'm not sure any mail program uses it.
dmurray 5y ago
> I cannot set a single lang: attribute for the whole page because it'll be wrong for the other two languages. > What does that leave me? A dropdown list "C/J/K/Other" besides every single text field? If you need to control the language display at the granularity of a single text field (rather than a user or a page or a website), then yes, you need a tool that operates on a single text field. This shouldn't be too surprising. Surely you can get away with one of your other solutions, though. In particular, you can guess the language and be right most of the time.
- dvdkon 5y ago
  It is surprising to me, because I can mix Czech, English and German just fine on a single screen, even in a single text field. From that perspective having to say "this whole text is in language X" seems backwards.
  - dmurray 5y ago
    OK, yeah. If Unicode hadn't gone with Han unification (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) you'd have the same flexibility in mixing C/J/K.
    - rectang 5y ago
      > (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) Representation matters. There should have been plenty of experts who understood that Han unification was problematic. It seems they were not in a position to do anything about it.
      - masklinn 5y ago
        Part of the problem back then is they really really wanted to fit it all in 16 bits, and given there are more hanzi than that to start with this was a bit of an issue (back in the 80s the largest dictionary listed >54000 characters, now it’s >100000, and Japanese and Korean total about 50000 each). A few years later they increased the character space by 5 bits and it wasn’t an issue anymore, but the original legacy of han unification remains.
makeitdouble 5y ago
For what it's worth, mainstream OSes will also have poor handling of these cases, which will eliminate the most tricky cases by the sheer inconvenience it causes. As far as I know textfields only have one font applied, so entering both languages in a single field won't be optimal. And if you're not doing anything fancy with your fields, they will all take the same font as well. So even at the input level, the user switching languages will already be mildly screwed, and the best solution would probably be to change pages for each language.
deleted 5y ago
[deleted]
Asooka 5y ago
Trying to do something smart is usually the wrong approach and leads to users tangling themselves in invisible state that they do not understand and can't change. The best thing would have been for Unicode to not do Han unification. The second best would be to provide alternate glyphs now. The third best is to display the characters either in the language they're written in, when you know that for sure (usually for text that you wrote yourself), or in the user's most likely locale, when you don't. For locale I would go down this list of traits and pick the first that matches: 1) The language setting on your website if you have one and have translated it to C/J/K. You may use different TLDs for the different languages and discern that way, too. 2) The list of preferred languages from the browser. This is usually unreliable, but if someone has gone to the trouble of inputting "english=1;japanese=.9;chinese=.8", then it's a fair bet they want Japanese Kanji usually and will be understanding if you use them in place of Chinese Han characters. 3) The country to which the user's IP belongs. The least ideal option, but if you're in Korea and reading a random string of Hanzi, you probably expect them to look like Hanzi. You will show the wrong characters to some users, but the behaviour is understandable. "Oh, the site is showing me Korean characters because I'm in Korea." is a lot easier to grasp than "The site is showing me Chinese characters because I clicked a dropdown one time that I forgot about and now I have no idea why my name is written wrong!" You can argue about point 2) that some users might set their language preferences and forget about it, but so far I have never observed a user who doesn't know about them messing with the setting.
gpderetta 5y ago
Deprecate languages. We will all be writing exclusively in emojis from now on.
- account42 5y ago
  ツ
dr-detroit 5y ago
This problem shows up in email too. The internet is mostly hollow and broken.
GoblinSlayer 5y ago
>How is a Japanese user asking for help learning Chinese supposed to write? They write with japanese kanji. I imagine minuscule differences in kanji forms are negligible compared to general unfamiliarity with the foreign language.
deleted 5y ago
[deleted]

zokier 5y ago

> unix time is non-monotonic and discontinuous at random Well, depends on your definition of unix time; if you use time() as the definition then it is actually monotonic because the integral part only repeats on leap seconds?

dathinab 5y ago

The problem is btw. not specific to Han unification/asian languages. It's that for every western language if you use screen readers.

the_other 5y ago
My thought whilst reading the article was that the Han unification would actually help screen readers. IIUC The meaning of the glyph is the same across all the languages, so the screen reader will get the correct meaning and can present it according to local settings. The problem with the European languages is that the different characters (letter variations, accent variations) can change the meaning of the word they're part of. Or have I misunderstood?
- lifthrasiir 5y ago
  The same Han character can have wildly different pronunciations even in a single language (being a logogram, they represent a word and not a sound). KS X 1001, the primary Korean character set, even duplicated same characters according to their readings so that they can be almost [1] correctly converted back to Hangul. In practice they didn't work well though, and Unicode assigns all but one duplicate characters into the compatibility region. [1] These readings didn't take account for systematic variations like the initial sound law (두음법칙, for example 이 vs. 리 at the beginning of words).
- soraminazuki 5y ago
  You can't split CJK sentences into individual letters, inspect them one by one and decipher its exact meaning. If you present Chinese writing to a Japanese speaker, they'd only see complete gibberish consisting of letters they may or may not recognize. It also works the other way around too. On top of that, Kanjis aren't the only type of letters Japanese folks use. They also use Hiraganas and Katakanas, which are phonetic symbols and totally unrecognizable to non-Japanese speakers.
Snild 5y ago
I'm not sure I understand what you're saying here. Are you saying that screen readers don't know which language to interpret text as?
- dathinab 5y ago
  If you don't tell them, yes. As many websites do not convey the language properly they try to fix that using some heuristics, but especially for small texts (like messages) that can easily fail hard leading to completely wrong pronunciation. Mail for example tried to fix that by allowing you to annotate the language for UTF-8 embedded in mail headers and the content. Ironically while that mechanism works for display names it doesn't work for email addresses itself. And I'm not sure any mail program uses it.

dmurray 5y ago

> I cannot set a single lang: attribute for the whole page because it'll be wrong for the other two languages. > What does that leave me? A dropdown list "C/J/K/Other" besides every single text field? If you need to control the language display at the granularity of a single text field (rather than a user or a page or a website), then yes, you need a tool that operates on a single text field. This shouldn't be too surprising. Surely you can get away with one of your other solutions, though. In particular, you can guess the language and be right most of the time.

dvdkon 5y ago
It is surprising to me, because I can mix Czech, English and German just fine on a single screen, even in a single text field. From that perspective having to say "this whole text is in language X" seems backwards.
- dmurray 5y ago
  OK, yeah. If Unicode hadn't gone with Han unification (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) you'd have the same flexibility in mixing C/J/K.
  - rectang 5y ago
    > (couldn't they know we'd have space to put zillions of characters in a font just a few years later?) Representation matters. There should have been plenty of experts who understood that Han unification was problematic. It seems they were not in a position to do anything about it.
    - masklinn 5y ago
      Part of the problem back then is they really really wanted to fit it all in 16 bits, and given there are more hanzi than that to start with this was a bit of an issue (back in the 80s the largest dictionary listed >54000 characters, now it’s >100000, and Japanese and Korean total about 50000 each). A few years later they increased the character space by 5 bits and it wasn’t an issue anymore, but the original legacy of han unification remains.

makeitdouble 5y ago

For what it's worth, mainstream OSes will also have poor handling of these cases, which will eliminate the most tricky cases by the sheer inconvenience it causes. As far as I know textfields only have one font applied, so entering both languages in a single field won't be optimal. And if you're not doing anything fancy with your fields, they will all take the same font as well. So even at the input level, the user switching languages will already be mildly screwed, and the best solution would probably be to change pages for each language.

deleted 5y ago

[deleted]

Asooka 5y ago

Trying to do something smart is usually the wrong approach and leads to users tangling themselves in invisible state that they do not understand and can't change. The best thing would have been for Unicode to not do Han unification. The second best would be to provide alternate glyphs now. The third best is to display the characters either in the language they're written in, when you know that for sure (usually for text that you wrote yourself), or in the user's most likely locale, when you don't. For locale I would go down this list of traits and pick the first that matches: 1) The language setting on your website if you have one and have translated it to C/J/K. You may use different TLDs for the different languages and discern that way, too. 2) The list of preferred languages from the browser. This is usually unreliable, but if someone has gone to the trouble of inputting "english=1;japanese=.9;chinese=.8", then it's a fair bet they want Japanese Kanji usually and will be understanding if you use them in place of Chinese Han characters. 3) The country to which the user's IP belongs. The least ideal option, but if you're in Korea and reading a random string of Hanzi, you probably expect them to look like Hanzi. You will show the wrong characters to some users, but the behaviour is understandable. "Oh, the site is showing me Korean characters because I'm in Korea." is a lot easier to grasp than "The site is showing me Chinese characters because I clicked a dropdown one time that I forgot about and now I have no idea why my name is written wrong!" You can argue about point 2) that some users might set their language preferences and forget about it, but so far I have never observed a user who doesn't know about them messing with the setting.

gpderetta 5y ago

Deprecate languages. We will all be writing exclusively in emojis from now on.

account42 5y ago
ツ

dr-detroit 5y ago

This problem shows up in email too. The internet is mostly hollow and broken.

GoblinSlayer 5y ago

>How is a Japanese user asking for help learning Chinese supposed to write? They write with japanese kanji. I imagine minuscule differences in kanji forms are negligible compared to general unfamiliarity with the foreign language.

deleted 5y ago

[deleted]