4 ms·

No, Unicode fares terribly for CJK languages, see Han Unification. I can't stress enough how it is a lie that Unicode can represent all languages in use, when i

No, Unicode fares terribly for CJK languages, see Han Unification. I can't stress enough how it is a lie that Unicode can represent all languages in use, when it cannot even distinguish Japanese from Chinese.

jcranmer 9y ago
It can't distinguish between English, French, German, Swedish, Norwegian, Spanish, etc. either. And there's no charset that does that. Do you find that a problem? If so, you're very much in the minority; if not, why distinguish between Japanese and Chinese but not English and French?
- kalleboo 9y ago
  Because the characters are different enough that it causes actual day-to-day problems (I see text all the time on printouts/signs where some incorrect font substitution has taken place and you get Chinese instead of Japanese characters). You can (probably correctly) argue that these are really just deficiencies in text editing/markup tools where you can't mark the language of text but the fact that that's required for correct presentation only in CJK languages seems to indicate that it's a problem with this unification in particular. How would you like if there was Latin/Greek/Cyrillic unification? I mean they're all just alphabets that make the same sounds they're all basically the same right?
  - mixmastamyk 9y ago
    Wouldn't bother me personally, why should it? However, as there are only a few tens of characters between them, it wouldn't save much.
- jack1243star 9y ago
  Swedish ä and German ä are the same character. 令 is a different character (of the same origin) in Japanese, Traditional Chinese and Simplied Chinese (and it depends on your browser setup which you get on screen, which is absolutely insane). It is like saying we should use English p as Cyrillic р, which only look similar, or as п, because they have the same origin or sound? And no, CJK language users is not a minority, although many of us have become used to the font inconsistensies (because while wrong, they are legible to us).
  - jcranmer 9y ago
    The question comes down to if Chinese hanzi and Japanese kanji are the same alphabet or not. It's not clear what the answer should be, but for Unicode, it was absolutely necessary to treat them as the same alphabet, if they were not to break the 64K character limit (it's dubious Unicode would have seen its modern universality if it weren't a compact fixed 16-bit encoding, especially since the genius of UTF-8 encoding was a rather later addition). Another point that is commonly missed is that Han unification was first proposed by East Asians (not ignorant Westerners), specifically the Chinese (which is why the Chinese don't really object to Unicode but the Japanese do). A side note: it's interesting that you bring up ä as an example, because ä is actually a typographic unification of two very distinct letters. In languages like English and French, the diacritic is a diaeresis: a mark, derived from Ancient Greek indicating that the vowel is to be pronounced separately rather than as a diphthong. In languages like German, it's an umlaut, where it's a typographic reduction of a superscript e. They are in fact very different characters that merely look the same.
    - jack1243star 9y ago
      Whether the ideographs are the "same" or not is definitely a can of worm, both academically and politically speaking. I am aware of the limits that the original Unicode specs were subjected to, which adds to my grief on how things could have been. Now that it has taken over the world we have to deal with wrong characters displayed for many applications. (Even modern day iOS has problems displaying mixed language content in system text fields.) I don't know much about the history of western languages, so I cannot comment on the issue, just that it feels unjustified that during development CJK languages got crammed together and messed up (no matter who did it). More and more emojis and other symbols just rub salt into the wound.
      - mixmastamyk 9y ago
        I think it makes sense to "compress" the CJK characters to points where their appearance is the same, because there are on the order of one hundred thousand of them. For western alphabets that have thirty or so symbols, the need is not nearly so acute.

jcranmer 9y ago

It can't distinguish between English, French, German, Swedish, Norwegian, Spanish, etc. either. And there's no charset that does that. Do you find that a problem? If so, you're very much in the minority; if not, why distinguish between Japanese and Chinese but not English and French?

kalleboo 9y ago
Because the characters are different enough that it causes actual day-to-day problems (I see text all the time on printouts/signs where some incorrect font substitution has taken place and you get Chinese instead of Japanese characters). You can (probably correctly) argue that these are really just deficiencies in text editing/markup tools where you can't mark the language of text but the fact that that's required for correct presentation only in CJK languages seems to indicate that it's a problem with this unification in particular. How would you like if there was Latin/Greek/Cyrillic unification? I mean they're all just alphabets that make the same sounds they're all basically the same right?
- mixmastamyk 9y ago
  Wouldn't bother me personally, why should it? However, as there are only a few tens of characters between them, it wouldn't save much.
jack1243star 9y ago
Swedish ä and German ä are the same character. 令 is a different character (of the same origin) in Japanese, Traditional Chinese and Simplied Chinese (and it depends on your browser setup which you get on screen, which is absolutely insane). It is like saying we should use English p as Cyrillic р, which only look similar, or as п, because they have the same origin or sound? And no, CJK language users is not a minority, although many of us have become used to the font inconsistensies (because while wrong, they are legible to us).
- jcranmer 9y ago
  The question comes down to if Chinese hanzi and Japanese kanji are the same alphabet or not. It's not clear what the answer should be, but for Unicode, it was absolutely necessary to treat them as the same alphabet, if they were not to break the 64K character limit (it's dubious Unicode would have seen its modern universality if it weren't a compact fixed 16-bit encoding, especially since the genius of UTF-8 encoding was a rather later addition). Another point that is commonly missed is that Han unification was first proposed by East Asians (not ignorant Westerners), specifically the Chinese (which is why the Chinese don't really object to Unicode but the Japanese do). A side note: it's interesting that you bring up ä as an example, because ä is actually a typographic unification of two very distinct letters. In languages like English and French, the diacritic is a diaeresis: a mark, derived from Ancient Greek indicating that the vowel is to be pronounced separately rather than as a diphthong. In languages like German, it's an umlaut, where it's a typographic reduction of a superscript e. They are in fact very different characters that merely look the same.
  - jack1243star 9y ago
    Whether the ideographs are the "same" or not is definitely a can of worm, both academically and politically speaking. I am aware of the limits that the original Unicode specs were subjected to, which adds to my grief on how things could have been. Now that it has taken over the world we have to deal with wrong characters displayed for many applications. (Even modern day iOS has problems displaying mixed language content in system text fields.) I don't know much about the history of western languages, so I cannot comment on the issue, just that it feels unjustified that during development CJK languages got crammed together and messed up (no matter who did it). More and more emojis and other symbols just rub salt into the wound.
    - mixmastamyk 9y ago
      I think it makes sense to "compress" the CJK characters to points where their appearance is the same, because there are on the order of one hundred thousand of them. For western alphabets that have thirty or so symbols, the need is not nearly so acute.