4 ms·

Han unification has been overly aggressive about merging some characters, but the basic principle is not as flawed as it is some times (as in this article) made

by frivoal 12y ago

Han unification has been overly aggressive about merging some characters, but the basic principle is not as flawed as it is some times (as in this article) made to sound. The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. Sometimes, when characters are a bit different, it is a matter of font. Out of the tends of thousands of characters used by these languages, it is not surprising that mistakes have been made, and yes, han unification has been a bit too far reaching. But not unifying at all would have been much worse. You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. Also, since CJK languages are comfortable with having an unbounded set of letters to work from, they tend to be comfortable with declaring every odd thing a separate letter. Witness the mess that half-width katakana is. Or notice that the origin of most emoji is not in the disregard of white old men for "lesser" languages, but comes from importing existing characters from Japanese encodings. Which also hints at one more factor. Historically, and still very much to this day, Japan has been much more active in standardization bodies than most other non western countries. Unsurprisingly, Japanese is much better supported by modern software than many other non western languages.

com2kid 12y ago
> The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. And what about traditional versus simplified? Which glyph set do I use? Oh wait, thanks to Han unification, I now need to rely on bloody environment variables to decide! For Chinese text, rendering a string of text involves not just the string, but also the local runtime. Joy of joys. > You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. If following the spec means my users become partially illiterate then there are problems with the spec.
- Tomte 12y ago
  Antiqua, Fractura, Schwabacher, Textura and all those other charming variants of writing European languages don't have seperate code points for all those presentational variations. Do we white Western men happen to discriminate against ourselves?
  - com2kid 12y ago
    Those are all stylistic differences. You can still read any of those fonts. Sure some designer can go overboard and make a font hard to read, but that is just a designer being overly fancy! Think instead of the difference between Cyrillic and Latin. Sure if you squint hard enough they both have a common origin (Greek), but you'd be rather annoyed if you setup your phone, selected "English" and the OS used Cyrillic letters to spell out English words. Likewise you'd be annoyed if Greek letters were used. And you'd be even more upset if some products decided to use Cyrillic, some Greek, and some used Latin.
    - Tomte 12y ago
      Actually, I can't. I can decipher quite a bit with lots of effort, but I wouldn't call that "reading". And there are very few people around who can read all those (especially Textura) without problems. On your other example: when I was in Russia, I found those "unknown" letters difficult, but fun. But the letters they share with Latin? I didn't see a difference. And to your phone example: of course, I'd be annoyed. But that's exactly the point: you have to set up your local system correctly, to your expectations and standards. I'd be much more annoyed if some web page showed an English text in some Chinese transliteration, just because the author (writing English!) was Chinese. But that's basically what you propose! Sorry, I truly think you've run up an argumentative dead end.
      - com2kid 12y ago
        > On your other example: when I was in Russia, I found those "unknown" letters difficult, but fun. But the letters they share with Latin? I didn't see a difference. Again, Han unification does this. Most characters are the same, great! But some are different. Sucks for those that are different. > And to your phone example: of course, I'd be annoyed. But that's exactly the point: you have to set up your local system correctly, to your expectations and standards. The problem here is that for almost everything else, Unicode is a mapping from a code point (or a set of code points) to some distinct and unique representation on screen. Except there are some code points for which that isn't true. > Sorry, I truly think you've run up an argumentative dead end. You are not arguing any point, other than "let's keep this historical attempt at saving on encoding space around, even though we have expanded out the encoding space so that we don't need the savings anymore." That isn't a strong argument. My argument is "user's don't like this, it upsets them, we shouldn't do it." If something we do as engineers angers or upsets our users, we are doing it wrong. Flat out.
amyjess 12y ago
> You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. A much more apt comparison would be Antiqua and Fraktur. It was a commonly-held belief that Fraktur was the authentic German alphabet, separate from the alphabets used by other languages. Back in the 19th century, English-German dictionaries even used Antiqua for English words and Fraktur for German words.
- frivoal 12y ago
  no, it isn't a more apt comparison. Antiqua and Fraktur look much more different from each other than most Chinese and Japanese characters are from eachother. Here are two screenshots in Japanese and Chinese, taken from the http://www.yomiuri.co.jp/ http://www.yomiuri.co.jp/ for one, and http://www.cntv.cn/ http://www.cntv.cn/ for the other. Neither media outlet can be accused of being cultural sellouts misrepresenting the written tradition of their countries. Japanese: https://www.dropbox.com/s/lpb3dk26ftds6oh/Screenshot%202015-03-17%2022.39.39.png?dl=0 https://www.dropbox.com/s/lpb3dk26ftds6oh/Screenshot%202015-... Chinese: https://www.dropbox.com/s/dcwv33ho0juracg/Screenshot%202015-03-17%2022.39.50.png?dl=0 https://www.dropbox.com/s/dcwv33ho0juracg/Screenshot%202015-... These are the same characters, and unifying them is the only sane thing to do. The same is true for most characters. There are also some characters which are clearly distinct in both scripts, and unicode treats them as different. Some are grey-zone, and there is room for reasonable disagreement. But as a whole, the case of unification of Chinese and Japanese is stronger than for Antiqua and Fraktur.
  - kalleboo 12y ago
    Your screenshots are showing the exact same glyphs because you don't have the "宋体" font installed that's specified by the Chinese web page (it's a script font so you'd notice instantly it's different), so your browser picked a Japanese font to render it instead. The only reason they look the same in your screenshots is due to Han Unification. Your reasoning is "they're unified in Unicode which is proof they should be unified in Unicode".
    - 12y ago

by frivoal 12y ago

com2kid 12y ago

> The vast majority of Japanese and Chinese characters are not only similar, they are identical. Not all are. Some are clearly different characters deriving from a common historical root, and should not be unified. And what about traditional versus simplified? Which glyph set do I use? Oh wait, thanks to Han unification, I now need to rely on bloody environment variables to decide! For Chinese text, rendering a string of text involves not just the string, but also the local runtime. Joy of joys. > You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. If following the spec means my users become partially illiterate then there are problems with the spec.

Tomte 12y ago
Antiqua, Fractura, Schwabacher, Textura and all those other charming variants of writing European languages don't have seperate code points for all those presentational variations. Do we white Western men happen to discriminate against ourselves?
- com2kid 12y ago
  Those are all stylistic differences. You can still read any of those fonts. Sure some designer can go overboard and make a font hard to read, but that is just a designer being overly fancy! Think instead of the difference between Cyrillic and Latin. Sure if you squint hard enough they both have a common origin (Greek), but you'd be rather annoyed if you setup your phone, selected "English" and the OS used Cyrillic letters to spell out English words. Likewise you'd be annoyed if Greek letters were used. And you'd be even more upset if some products decided to use Cyrillic, some Greek, and some used Latin.
  - Tomte 12y ago
    Actually, I can't. I can decipher quite a bit with lots of effort, but I wouldn't call that "reading". And there are very few people around who can read all those (especially Textura) without problems. On your other example: when I was in Russia, I found those "unknown" letters difficult, but fun. But the letters they share with Latin? I didn't see a difference. And to your phone example: of course, I'd be annoyed. But that's exactly the point: you have to set up your local system correctly, to your expectations and standards. I'd be much more annoyed if some web page showed an English text in some Chinese transliteration, just because the author (writing English!) was Chinese. But that's basically what you propose! Sorry, I truly think you've run up an argumentative dead end.
    - com2kid 12y ago
      > On your other example: when I was in Russia, I found those "unknown" letters difficult, but fun. But the letters they share with Latin? I didn't see a difference. Again, Han unification does this. Most characters are the same, great! But some are different. Sucks for those that are different. > And to your phone example: of course, I'd be annoyed. But that's exactly the point: you have to set up your local system correctly, to your expectations and standards. The problem here is that for almost everything else, Unicode is a mapping from a code point (or a set of code points) to some distinct and unique representation on screen. Except there are some code points for which that isn't true. > Sorry, I truly think you've run up an argumentative dead end. You are not arguing any point, other than "let's keep this historical attempt at saving on encoding space around, even though we have expanded out the encoding space so that we don't need the savings anymore." That isn't a strong argument. My argument is "user's don't like this, it upsets them, we shouldn't do it." If something we do as engineers angers or upsets our users, we are doing it wrong. Flat out.

amyjess 12y ago

> You don't want a in German and a in English to be different letters just because Helvetica and Baskerville look different. A much more apt comparison would be Antiqua and Fraktur. It was a commonly-held belief that Fraktur was the authentic German alphabet, separate from the alphabets used by other languages. Back in the 19th century, English-German dictionaries even used Antiqua for English words and Fraktur for German words.

frivoal 12y ago
no, it isn't a more apt comparison. Antiqua and Fraktur look much more different from each other than most Chinese and Japanese characters are from eachother. Here are two screenshots in Japanese and Chinese, taken from the http://www.yomiuri.co.jp/ http://www.yomiuri.co.jp/ for one, and http://www.cntv.cn/ http://www.cntv.cn/ for the other. Neither media outlet can be accused of being cultural sellouts misrepresenting the written tradition of their countries. Japanese: https://www.dropbox.com/s/lpb3dk26ftds6oh/Screenshot%202015-03-17%2022.39.39.png?dl=0 https://www.dropbox.com/s/lpb3dk26ftds6oh/Screenshot%202015-... Chinese: https://www.dropbox.com/s/dcwv33ho0juracg/Screenshot%202015-03-17%2022.39.50.png?dl=0 https://www.dropbox.com/s/dcwv33ho0juracg/Screenshot%202015-... These are the same characters, and unifying them is the only sane thing to do. The same is true for most characters. There are also some characters which are clearly distinct in both scripts, and unicode treats them as different. Some are grey-zone, and there is room for reasonable disagreement. But as a whole, the case of unification of Chinese and Japanese is stronger than for Antiqua and Fraktur.
- kalleboo 12y ago
  Your screenshots are showing the exact same glyphs because you don't have the "宋体" font installed that's specified by the Chinese web page (it's a script font so you'd notice instantly it's different), so your browser picked a Japanese font to render it instead. The only reason they look the same in your screenshots is due to Han Unification. Your reasoning is "they're unified in Unicode which is proof they should be unified in Unicode".
  - 12y ago