4 ms·

Unicode's Han unification makes this more difficult than it needs to be. Now that Unicode has more than 16 bits worth of characters anyway, it looks very much l

by mrob 8y ago

Unicode's Han unification makes this more difficult than it needs to be. Now that Unicode has more than 16 bits worth of characters anyway, it looks very much like a mistake. https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification

jessaustin 8y ago
One can kind of understand why they did it. A lot of glyphs are the same from one language to another. If the codepoint for Chinese "大" were different from that for the Japanese "大" were different from that for the Korean "大", that would generate some complaints. OTOH, a Unicode that includes both "学" and "學" certainly has enough room for several versions of "道". I suspect this will eventually be fixed, but as a series of one-off additional codepoints, not as a giant duplication for each of C, J, K, and V. Some of the problem seems to have been that while the Chinese Unicode experts realized they wanted to write both "学" and "學" in the same document, the Korean experts never considered the possibility of including Chinese or Japanese text in a primarily Korean document. I think by this point they've realized it, though, so it will be fixed eventually.
- kevin_thibedeau 8y ago
  The consortium always expected implementations to use higher level markup to specify language specific variants.
  - Dylan16807 8y ago
    That can work in an unwieldy way for entire documents. But for inside a single document it should absolutely be Unicode's job to prevent quasi-mojibake.
  - mrob 8y ago
    That's impossible in some cases. How am I supposed to list a directory with a mix of Chinese and Japanese filenames? Most filesystems don't have a filename language attribute.
  - jessaustin 8y ago
    "The consortium" is not a monolith. Some of them probably did expect that, but such expectation seems to conflict with the "学"/"學" distinction mentioned above and for that matter with similar distinctions for other scripts e.g. "a"/"α" and "A"/"Α".

by mrob 8y ago

jessaustin 8y ago

One can kind of understand why they did it. A lot of glyphs are the same from one language to another. If the codepoint for Chinese "大" were different from that for the Japanese "大" were different from that for the Korean "大", that would generate some complaints. OTOH, a Unicode that includes both "学" and "學" certainly has enough room for several versions of "道". I suspect this will eventually be fixed, but as a series of one-off additional codepoints, not as a giant duplication for each of C, J, K, and V. Some of the problem seems to have been that while the Chinese Unicode experts realized they wanted to write both "学" and "學" in the same document, the Korean experts never considered the possibility of including Chinese or Japanese text in a primarily Korean document. I think by this point they've realized it, though, so it will be fixed eventually.

kevin_thibedeau 8y ago
The consortium always expected implementations to use higher level markup to specify language specific variants.
- Dylan16807 8y ago
  That can work in an unwieldy way for entire documents. But for inside a single document it should absolutely be Unicode's job to prevent quasi-mojibake.
- mrob 8y ago
  That's impossible in some cases. How am I supposed to list a directory with a mix of Chinese and Japanese filenames? Most filesystems don't have a filename language attribute.
- jessaustin 8y ago
  "The consortium" is not a monolith. Some of them probably did expect that, but such expectation seems to conflict with the "学"/"學" distinction mentioned above and for that matter with similar distinctions for other scripts e.g. "a"/"α" and "A"/"Α".