6 ms·

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification https://en.m.wikipedia.org/wiki/Han_unification As

by shpx 7y ago

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk about it.

gsnedders 7y ago
Relatively few people frequently look at different Han languages, and relatively few people are looking at txt files containing Han characters (and I expect those that do are typically running with their OS locale set to one of the Han languages?). Enough CJK HTML content is tagged and heuristics are mostly good enough that incorrect font selection isn't a massive issue on the web, and AFAIK most major word processors include metadata in the file that suffices to distinguish language.
- Grue3 7y ago
  It's a massive issue on web, I routinely see Japanese posts on Twitter displayed in a Chinese font when they consist only of kanji.
klodolph 7y ago
I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”, there might be between one and three different Unicode characters, and there might be between one and three different ways of writing it. Here’s an illustration: https://japanese.stackexchange.com/questions/64590/why-are-japanese-fonts-different-to-chinese https://japanese.stackexchange.com/questions/64590/why-are-j... So the issue does come up when mixing Chinese and Japanese text, but it’s not really one that has a big impact on legibility of the text but you would definitely be concerned if you were writing a Japanese textbook for Chinese students, or vice versa. Beyond that, it is usually fairly trivial to distinguish between Japanese and Chinese text, so you could just lean on simple heuristics to get the work done (Japanese text, with the exception of fairly ancient text or very short fragments, contains kana, but Chinese does not).
- cryptonector 7y ago
  As I understand it Han unification happened because at the time all there was was UCS-2 -no UTF-16, no UTF-8- so codespace was tight and precious, and that motivated codespace preserving optimizations, of which Han unification is the notable one. To avoid that they needed to have invented UTF-8 many years earlier. Perhaps if the people designing UTF-8 were more diverse they might have felt the necessity to invent UTF-8 to the point of actually doing it, but then perhaps they might have done it poorly. At any rate, I don't know enough details to really know if "Han unification was pushed through by western interests" is remotely fair.
  - macintux 7y ago
    UTF-8 was sketched on a placemat as a response to a different idea. It seems likely that had it not arisen in a moment of inspiration by a genius, we would be stuck with another inferior design by committee. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
    - gpderetta 7y ago
      Utf-8 is a magnificent hack.
    - cryptonector 7y ago
      I agree. But too, necessity is the mother of invention. GP seems to argue that Han unification happened because the UC was not diverse enough. Maybe, and maybe if it had been diverse enough the need would have arisen sooner. But again, the thing they came up with could have been garbage, who knows! What I do know is that UTF-8 is genius. The Han unification problems seem mostly minor -- I suspect code can detect language and do the right thing, for example, and again, we could revive language tags if need be.
- cygx 7y ago
  Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode German Fraktur: While some characters are available due to their use in mathematics, it's lacking the corresponding variants of ä, ö, ü, ß, ſ as well as specific ligatures. So if you want to intermix modern with old German writing, you'll also have to go out-of-band.
cryptonector 7y ago
First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode for doing away with Han unification, which is why no one mentions it: besides crying about it, what else can one do? Maybe we should revive language tags? Anyways, isn't the difference between unified Han/Kanji characters mostly stylistic rather than semantic? I'm not denying that many users would get annoyed, but again, what to do about it??
- innocenat 7y ago
  It's only stylistic issue if you also consider a and α (alpha) to also be just stylistic different. I have learned to live with it, but it is very annoying.
  - cryptonector 7y ago
    Do you want to see all Han texts in your preferred style, or do you want to see them in the style corresponding to the language they were written in?
    - innocenat 7y ago
      In the original language, of course. Why is that even a question. That is like asking if Geek people would want to read Latin script using Greek alphabet or not. You keep using the word 'style', so you agree that α is a style of a? Then I have no more comment. It's not 'style' at all.
      - cryptonector 7y ago
        > Why is that even a question. Why wouldn't it be. As I'm not a user of CJK scripts it seemed like a fair question to ask. > You keep using the word 'style', so you agree that α is a style of a? I didn't have a better word for it. Oddly alpha is getting different renderings in the text input box than in your comment. So Han unification makes CJK as hard to read for you as mixing Greek and Latin scripts would for me. That's what I wanted to understand. Thanks.

by shpx 7y ago

gsnedders 7y ago

Relatively few people frequently look at different Han languages, and relatively few people are looking at txt files containing Han characters (and I expect those that do are typically running with their OS locale set to one of the Han languages?). Enough CJK HTML content is tagged and heuristics are mostly good enough that incorrect font selection isn't a massive issue on the web, and AFAIK most major word processors include metadata in the file that suffices to distinguish language.

Grue3 7y ago
It's a massive issue on web, I routinely see Japanese posts on Twitter displayed in a Chinese font when they consist only of kanji.

klodolph 7y ago

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”, there might be between one and three different Unicode characters, and there might be between one and three different ways of writing it. Here’s an illustration: https://japanese.stackexchange.com/questions/64590/why-are-japanese-fonts-different-to-chinese https://japanese.stackexchange.com/questions/64590/why-are-j... So the issue does come up when mixing Chinese and Japanese text, but it’s not really one that has a big impact on legibility of the text but you would definitely be concerned if you were writing a Japanese textbook for Chinese students, or vice versa. Beyond that, it is usually fairly trivial to distinguish between Japanese and Chinese text, so you could just lean on simple heuristics to get the work done (Japanese text, with the exception of fairly ancient text or very short fragments, contains kana, but Chinese does not).

cryptonector 7y ago
As I understand it Han unification happened because at the time all there was was UCS-2 -no UTF-16, no UTF-8- so codespace was tight and precious, and that motivated codespace preserving optimizations, of which Han unification is the notable one. To avoid that they needed to have invented UTF-8 many years earlier. Perhaps if the people designing UTF-8 were more diverse they might have felt the necessity to invent UTF-8 to the point of actually doing it, but then perhaps they might have done it poorly. At any rate, I don't know enough details to really know if "Han unification was pushed through by western interests" is remotely fair.
- macintux 7y ago
  UTF-8 was sketched on a placemat as a response to a different idea. It seems likely that had it not arisen in a moment of inspiration by a genius, we would be stuck with another inferior design by committee. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
  - gpderetta 7y ago
    Utf-8 is a magnificent hack.
  - cryptonector 7y ago
    I agree. But too, necessity is the mother of invention. GP seems to argue that Han unification happened because the UC was not diverse enough. Maybe, and maybe if it had been diverse enough the need would have arisen sooner. But again, the thing they came up with could have been garbage, who knows! What I do know is that UTF-8 is genius. The Han unification problems seem mostly minor -- I suspect code can detect language and do the right thing, for example, and again, we could revive language tags if need be.
cygx 7y ago
Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode German Fraktur: While some characters are available due to their use in mathematics, it's lacking the corresponding variants of ä, ö, ü, ß, ſ as well as specific ligatures. So if you want to intermix modern with old German writing, you'll also have to go out-of-band.

cryptonector 7y ago

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode for doing away with Han unification, which is why no one mentions it: besides crying about it, what else can one do? Maybe we should revive language tags? Anyways, isn't the difference between unified Han/Kanji characters mostly stylistic rather than semantic? I'm not denying that many users would get annoyed, but again, what to do about it??

innocenat 7y ago
It's only stylistic issue if you also consider a and α (alpha) to also be just stylistic different. I have learned to live with it, but it is very annoying.
- cryptonector 7y ago
  Do you want to see all Han texts in your preferred style, or do you want to see them in the style corresponding to the language they were written in?
  - innocenat 7y ago
    In the original language, of course. Why is that even a question. That is like asking if Geek people would want to read Latin script using Greek alphabet or not. You keep using the word 'style', so you agree that α is a style of a? Then I have no more comment. It's not 'style' at all.
    - cryptonector 7y ago
      > Why is that even a question. Why wouldn't it be. As I'm not a user of CJK scripts it seemed like a fair question to ask. > You keep using the word 'style', so you agree that α is a style of a? I didn't have a better word for it. Oddly alpha is getting different renderings in the text input box than in your comment. So Han unification makes CJK as hard to read for you as mixing Greek and Latin scripts would for me. That's what I wanted to understand. Thanks.