8 ms·

“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native spea

“Whatever path we take, it’s imperative that the writing system of the 21st century be driven by the needs of the people using it. In the end, a non-native speaker – even one who is fluent in the language – cannot truly speak on behalf the monolingual, native speaker.” Not sure how the author can simultaneously say this, while criticizing the CJK unification, which makes total sense, and has never been a point of contention in the communities concerned with it. I regularly work with both Chinese and Japanese texts, and have studied Japanese language for more than five years; more than not opposing CJK unification, I benefit from it greatly. It means that I can search for the same character(yes, the same, I said it) and get results in Japanese, Chinese, and sometimes Korean context without having to consult some sort of unicode etymology resource. There should not be three codepoints for 中, 日, 力, and most other characters. The simplified and specific post-han forms have their own codepoints anyway, so there is(to my eyes at least) no value in fragmenting instances of the same character.

mehta 12y ago
> [...] CJK unification[...] has never been a point of contention in the communities concerned with it. I am not very familar with the CJK unification project so take my points with a grain of salt. > more than not opposing CJK unification, I benefit from it greatly. I think think that is a different point of view. Isn't it? You are seeing your benefit whereas the author is seeing his. Here's an alternative solution: What if the search engine understood what you were searching for and returned results in all the languages? Unification can result in a lot of information loss the same way a photo can be compressed but it comes at the cost of loss in quality. > so there is(to my eyes at least) no value in fragmenting instances of the same character. But no-one is fragmenting instances of the same character. They _are_ different characters from different languages. To take an example from the article, I am not sure how I feel about combining B and β. You are either ignoring the whole of English speaker population or the greek speaking one. Given that you have complete flexibility to assign a code for both of them, why not do it(responsibly)?
- Navarr 12y ago
  The example characters expressed: 日、中、力 however are written the same way in both Chinese and Japanese from my understanding. (Albeit, I studied Japanese). There are admittedly variations which should be done separately, however unification of visually identical glyphs is a "good thing" imho
  - frivoal 12y ago
    I'm fluent in Japanese and speak some Mandarin Chinese as well. These 3 characters are identical, not similar. For a different example, 国 and 國 used to be the same character, but China and Japan (left) have both diverged the traditional form still used in Taiwan (right). Unicode treats them as separate. 今 Looks slightly different in traditional Chinese vs other languages. In traditional Chinese, the little straight line between the two angled lines is sloped, while it is horizontal in simplified Chinese, Japanese or Korean. Any reader of any of these languages would have no issue if the variant they are used to was replaced by the other one. They might think you have a sloppy handwriting or an ugly font if they even notice, but that's about it. Unicode treats them as the same.
    - toufka 12y ago
      This seems again to be a perfect place for rendering rather than encoding. The english letter 'a' can be rendered as a ring with a tail (the way I handwrite), or a ring with a cap and a tail (the way the font usually renders). Both are the same letter, if rendered differently based on my (contextually sensitive) font.
      - ptaipale 12y ago
        I think not. I do want to be able to say both People's Republic of China 中华人民共和国 and Republic of China 中華民國 in the same text, and if I had to choose rendering either 国+华 or 國+華 then it wouldn't work.
        toufka 12y ago
ptx 12y ago
I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little. The arguments for Han unification could have just as well been applied to unifying the Nordic languages – the Swedish Ä is really exactly the same letter as Danish/Norwegian Æ (except that Swedish words never ever use the latter and presumably Danish words never use the former), so it could be argued that it should have the same codepoint, forcing us to use country-specific fonts and making it impossible to use both languages in a simple text file. So, for example, running "git log" on a project with an international contributor base would render either Swedish och Danish names incorrectly, depending on which font the user has selected. That would look extremely strange to me, so I imagine Chinese/Japanese/Korean users feel similarly. (Luckily for us Nordic users, our letters were already separate characters in ISO-8859-1, which Unicode is a superset of.)
- baddox 12y ago
  > I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little. That may be true, but not unifying at all must certainly fall under unifying too little.
- frivoal 12y ago
  Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. They are used for the same thing, they share etymology, but they are not the same letter. By the way, Unicode is about scripts, not languages. If we started distinguishing by language, we might need to start remembering that china doesn't have a single language. Duplicating all those characters again to cover Mandarin and Cantonese and Wu and Hakka and Hokkien? Should we have a different letter for Æ in Bokmål and Nynorsk? How about Riskmål? The equivalent of Æ and Ä in Han scripts have not been unified. The equivalent of a in Futura and a in Verdana have. There's tons of chinese characters, so a few of them are bound to be borderline and maybe contentious. But overall, Han unification was unavoidable for a project like Unicode.
  - TazeTSchnitzel 12y ago
    > Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. > They are used for the same thing, they share etymology, but they are not the same letter. That second quote applies equally to Ä and Æ.
jes5199 12y ago
I'm having trouble finding the reference now, but years ago I did read a story about how an early version of Unicode's reference document depicted CJK characters in a font with Chinese-styled characters, causing slowed adoption in Japan who had a gut reaction that "Unicode is too Chinese". Later editions printed several variations of characters, and the objections mostly evaporated. (Apologies if I've gotten the countries reversed in this story, it's been years and google is failing me)
evincarofautumn 12y ago
Not to say there aren’t problems with CJK unification. It’s difficult or impossible to represent old family names or newly coined characters without some means of composing characters from radicals—even if you do want “precomposed” characters a majority of the time, as is the case with, say, é (00E9) versus é (0065 0301). As far as these characters being “the same”, I think it’s better to say they’re analogues. A in English is analogous to A in Polish—both are Latin alphabets, but neither is the Latin alphabet, and these languages use “the same” letter quite differently. And, say, Å is a letter in Swedish, but in English it’s A with a separate accent mark. These are also just as well analogous to Α in Greek or А in Cyrillic, so hell, I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way.
- vorg 12y ago
  > I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way A 5 year old tech note by a Unicode Consortium member at http://www.unicode.org/notes/tn26 http://www.unicode.org/notes/tn26 gives 7 reasons "why the Latin, Greek, and Cyrillic scripts have been separately encoded, rather than being encoded as a single script", and 12 reasons why the Han script was unified. Edit: These 7 reasons make a good case why such "Greco Unification" wouldn't work.
  - deleted 12y ago
    [deleted]
  - evincarofautumn 12y ago
    Very good points. That’s why I said “if it could be done in a principled way”—I don’t think it can.
  - kibibu 12y ago
    I'm not convinced that this isn't just post-hoc justification. All seven reasons boil down to "It's always been done this way". I don't see how "a" in the Latin alphabet as used in English is any different to the "a" in the Latin alphabet used in Polish - in particular they are rendered in the same way. Yet the article claims that even though Chinese and Japanese versions of characters now appear wildly different they still represent the same script so bad luck. Should the Unicode consortium be concerned with the glyphs or the semantics? Seems they are selectively doing both.
eloisant 12y ago
Actually yes, the CJK unification is a problem for many people, including me when I want to read Japanese on a phone bought in Europe. Example 1. Typically, any time you want to mix the 2 languages you're getting in trouble. Let's say you write a textbook for Chinese people to learn Japanese as a second language. Or a research article in Japanese citing old Chinese literature. In your text, you'll have to mark specifically which part are in Japanese and which part are in Chinese and use different font for them. If you don't, the characters look wrong. Example 2. Most phone software don't switch font for languages, so they pick one. Let's say you buy a Japanese phone, it's a Japanese font everywhere. For some reason on a Samsung or Motorola phone bought in Europe, it's a Chinese font everywhere and it looks ugly for Japanese. Wasn't Unicode supposed to be universal, and allow you to read in any language with a device bought anywhere? On the other hand, I don't think the "I can't write my name" is really a problem for Japanese people, or at least it's not a Unicode problem. Some people have rare characters in their names. They accept and understand not being able to type their name on a computer and having to settle for a close characters. Unicode actually provided more characters than previous Japanese-only encodings like JIS so there are people who could write their name thanks to Unicode.
- krakensden 12y ago
  Neither of your examples are solveable by the Unicode consortium, who are not the god-emperors of fonts (nor would you want them to be).
  - mbessey 12y ago
    I respectfully disagree. If Japanese ideograms and Chinese ideograms actually used different code points (i.e. no "Han unification"), then the problem wouldn't exist - the phone could trivially use a Japanese font for Japanese text, and a Chinese font for Chinese text.
    - anon4 12y ago
      But.. you already posted a solution to your problem - use chinese font for chinese and japanese font for japanese. There is no problem. I mean, you would also presumably use a western font for latin and another separate font for cyrillic, since japanese fonts universally have absolutely dreadful kerning on latin (and often omit cyrillic entirely).
TazeTSchnitzel 12y ago
>There should not be three codepoints for 中, 日, 力, and most other characters There wouldn't be. Where the characters do not diverge, you don't need separate codepoints, surely?
wodenokoto 12y ago
I think it is a problem when 2 characters that are written differently share the same code point. Having 大 be the same code point for Chinese, Japanese and Korean is to me as obvious as having the same A for English and Italian. We agree on this point. But some of the unified characters listed on Wikipedia ought to have different code points because they are no longer written the same across languages. https://en.wikipedia.org/wiki/Han_unification#Examples_of_language_independent_characters https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... This is like having a and α share the same code point.
fenomas 12y ago
You're oversimplifying this. Take characters like 今 or 骨. Whether the C/J/K variants are the "same characters" or not is semantic, but philosophy aside they are written differently in each language. That is to say, if you are localizing an app into a CJK language, your app would not be working correctly if it displayed Japanese text with a Chinese 今, or vice versa. If you showed it to a user you'd get a bug report. But such characters have only one codepoint, so unless you have language metadata for your text, you can't render it correctly (which is half of was so bad about the pre-unicode days!). (Note: Unicode later added "variation selectors", which as far as I know solve the problems mentioned above. I don't know why the aren't in widespread use, or perhaps they are and I'm misunderstanding something.)
microcolonel 12y ago
I would like to take a moment to have people recognize that there are differences between the glyphs used for these characters, and some people find this to be a point of contention. For example, in the case of the text fragment “福祉”, there are three glyph representations for each character. Referenced below is an image demonstrating these three; first is Chinese, second is Korean, third is Japanese. [0] (I left out the Taiwanese etc. because they don't differ in this case from the CN) The UC have clear rules for this, as explained in their section "Characters, not glyphs", which starts on page 8 of the chapter 2 pdf of the Unicode 7.0 standard. They make the case that codepoints are not for representing stylistic or glyphic distinctions, as seen here, but rather for semantic separations; that is, people still agree that these are the same characters, even if they typeset and write them differently in Chinese, Korean, and Japanese text. They make cases specifically against a greco-latin unification, and for the CJK unification in Technical Note #26[1] [0] http://marumie.magnifi.ca:8080/ipfs/QmXDnJUPVtzxVtAQgFkNHwvkREL7DAKvgsLLEEWk9HAHNc/demonstration.svg http://marumie.magnifi.ca:8080/ipfs/QmXDnJUPVtzxVtAQgFkNHwvk... [1] http://www.unicode.org/notes/tn26/ http://www.unicode.org/notes/tn26/

mehta 12y ago

> [...] CJK unification[...] has never been a point of contention in the communities concerned with it. I am not very familar with the CJK unification project so take my points with a grain of salt. > more than not opposing CJK unification, I benefit from it greatly. I think think that is a different point of view. Isn't it? You are seeing your benefit whereas the author is seeing his. Here's an alternative solution: What if the search engine understood what you were searching for and returned results in all the languages? Unification can result in a lot of information loss the same way a photo can be compressed but it comes at the cost of loss in quality. > so there is(to my eyes at least) no value in fragmenting instances of the same character. But no-one is fragmenting instances of the same character. They _are_ different characters from different languages. To take an example from the article, I am not sure how I feel about combining B and β. You are either ignoring the whole of English speaker population or the greek speaking one. Given that you have complete flexibility to assign a code for both of them, why not do it(responsibly)?

Navarr 12y ago
The example characters expressed: 日、中、力 however are written the same way in both Chinese and Japanese from my understanding. (Albeit, I studied Japanese). There are admittedly variations which should be done separately, however unification of visually identical glyphs is a "good thing" imho
- frivoal 12y ago
  I'm fluent in Japanese and speak some Mandarin Chinese as well. These 3 characters are identical, not similar. For a different example, 国 and 國 used to be the same character, but China and Japan (left) have both diverged the traditional form still used in Taiwan (right). Unicode treats them as separate. 今 Looks slightly different in traditional Chinese vs other languages. In traditional Chinese, the little straight line between the two angled lines is sloped, while it is horizontal in simplified Chinese, Japanese or Korean. Any reader of any of these languages would have no issue if the variant they are used to was replaced by the other one. They might think you have a sloppy handwriting or an ugly font if they even notice, but that's about it. Unicode treats them as the same.
  - toufka 12y ago
    This seems again to be a perfect place for rendering rather than encoding. The english letter 'a' can be rendered as a ring with a tail (the way I handwrite), or a ring with a cap and a tail (the way the font usually renders). Both are the same letter, if rendered differently based on my (contextually sensitive) font.
    - ptaipale 12y ago
      I think not. I do want to be able to say both People's Republic of China 中华人民共和国 and Republic of China 中華民國 in the same text, and if I had to choose rendering either 国+华 or 國+華 then it wouldn't work.
      - toufka 12y ago

ptx 12y ago

I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little. The arguments for Han unification could have just as well been applied to unifying the Nordic languages – the Swedish Ä is really exactly the same letter as Danish/Norwegian Æ (except that Swedish words never ever use the latter and presumably Danish words never use the former), so it could be argued that it should have the same codepoint, forcing us to use country-specific fonts and making it impossible to use both languages in a simple text file. So, for example, running "git log" on a project with an international contributor base would render either Swedish och Danish names incorrectly, depending on which font the user has selected. That would look extremely strange to me, so I imagine Chinese/Japanese/Korean users feel similarly. (Luckily for us Nordic users, our letters were already separate characters in ISO-8859-1, which Unicode is a superset of.)

baddox 12y ago
> I agree, but it seems tricky like it would be tricky to strike exactly the right balance between unifying too much and too little. That may be true, but not unifying at all must certainly fall under unifying too little.
frivoal 12y ago
Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. They are used for the same thing, they share etymology, but they are not the same letter. By the way, Unicode is about scripts, not languages. If we started distinguishing by language, we might need to start remembering that china doesn't have a single language. Duplicating all those characters again to cover Mandarin and Cantonese and Wu and Hakka and Hokkien? Should we have a different letter for Æ in Bokmål and Nynorsk? How about Riskmål? The equivalent of Æ and Ä in Han scripts have not been unified. The equivalent of a in Futura and a in Verdana have. There's tons of chinese characters, so a few of them are bound to be borderline and maybe contentious. But overall, Han unification was unavoidable for a project like Unicode.
- TazeTSchnitzel 12y ago
  > Ä and Æ are more different from eachother than the characters in Chinese and Japanese which have been merged. > They are used for the same thing, they share etymology, but they are not the same letter. That second quote applies equally to Ä and Æ.

jes5199 12y ago

I'm having trouble finding the reference now, but years ago I did read a story about how an early version of Unicode's reference document depicted CJK characters in a font with Chinese-styled characters, causing slowed adoption in Japan who had a gut reaction that "Unicode is too Chinese". Later editions printed several variations of characters, and the objections mostly evaporated. (Apologies if I've gotten the countries reversed in this story, it's been years and google is failing me)

evincarofautumn 12y ago

Not to say there aren’t problems with CJK unification. It’s difficult or impossible to represent old family names or newly coined characters without some means of composing characters from radicals—even if you do want “precomposed” characters a majority of the time, as is the case with, say, é (00E9) versus é (0065 0301). As far as these characters being “the same”, I think it’s better to say they’re analogues. A in English is analogous to A in Polish—both are Latin alphabets, but neither is the Latin alphabet, and these languages use “the same” letter quite differently. And, say, Å is a letter in Swedish, but in English it’s A with a separate accent mark. These are also just as well analogous to Α in Greek or А in Cyrillic, so hell, I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way.

vorg 12y ago
> I would actually support the author’s “Greco Unification” strawman if it could be done in a principled way A 5 year old tech note by a Unicode Consortium member at http://www.unicode.org/notes/tn26 http://www.unicode.org/notes/tn26 gives 7 reasons "why the Latin, Greek, and Cyrillic scripts have been separately encoded, rather than being encoded as a single script", and 12 reasons why the Han script was unified. Edit: These 7 reasons make a good case why such "Greco Unification" wouldn't work.
- deleted 12y ago
  [deleted]
- evincarofautumn 12y ago
  Very good points. That’s why I said “if it could be done in a principled way”—I don’t think it can.
- kibibu 12y ago
  I'm not convinced that this isn't just post-hoc justification. All seven reasons boil down to "It's always been done this way". I don't see how "a" in the Latin alphabet as used in English is any different to the "a" in the Latin alphabet used in Polish - in particular they are rendered in the same way. Yet the article claims that even though Chinese and Japanese versions of characters now appear wildly different they still represent the same script so bad luck. Should the Unicode consortium be concerned with the glyphs or the semantics? Seems they are selectively doing both.

eloisant 12y ago

Actually yes, the CJK unification is a problem for many people, including me when I want to read Japanese on a phone bought in Europe. Example 1. Typically, any time you want to mix the 2 languages you're getting in trouble. Let's say you write a textbook for Chinese people to learn Japanese as a second language. Or a research article in Japanese citing old Chinese literature. In your text, you'll have to mark specifically which part are in Japanese and which part are in Chinese and use different font for them. If you don't, the characters look wrong. Example 2. Most phone software don't switch font for languages, so they pick one. Let's say you buy a Japanese phone, it's a Japanese font everywhere. For some reason on a Samsung or Motorola phone bought in Europe, it's a Chinese font everywhere and it looks ugly for Japanese. Wasn't Unicode supposed to be universal, and allow you to read in any language with a device bought anywhere? On the other hand, I don't think the "I can't write my name" is really a problem for Japanese people, or at least it's not a Unicode problem. Some people have rare characters in their names. They accept and understand not being able to type their name on a computer and having to settle for a close characters. Unicode actually provided more characters than previous Japanese-only encodings like JIS so there are people who could write their name thanks to Unicode.

krakensden 12y ago
Neither of your examples are solveable by the Unicode consortium, who are not the god-emperors of fonts (nor would you want them to be).
- mbessey 12y ago
  I respectfully disagree. If Japanese ideograms and Chinese ideograms actually used different code points (i.e. no "Han unification"), then the problem wouldn't exist - the phone could trivially use a Japanese font for Japanese text, and a Chinese font for Chinese text.
  - anon4 12y ago
    But.. you already posted a solution to your problem - use chinese font for chinese and japanese font for japanese. There is no problem. I mean, you would also presumably use a western font for latin and another separate font for cyrillic, since japanese fonts universally have absolutely dreadful kerning on latin (and often omit cyrillic entirely).

TazeTSchnitzel 12y ago

>There should not be three codepoints for 中, 日, 力, and most other characters There wouldn't be. Where the characters do not diverge, you don't need separate codepoints, surely?

wodenokoto 12y ago

I think it is a problem when 2 characters that are written differently share the same code point. Having 大 be the same code point for Chinese, Japanese and Korean is to me as obvious as having the same A for English and Italian. We agree on this point. But some of the unified characters listed on Wikipedia ought to have different code points because they are no longer written the same across languages. https://en.wikipedia.org/wiki/Han_unification#Examples_of_language_independent_characters https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... This is like having a and α share the same code point.

fenomas 12y ago

You're oversimplifying this. Take characters like 今 or 骨. Whether the C/J/K variants are the "same characters" or not is semantic, but philosophy aside they are written differently in each language. That is to say, if you are localizing an app into a CJK language, your app would not be working correctly if it displayed Japanese text with a Chinese 今, or vice versa. If you showed it to a user you'd get a bug report. But such characters have only one codepoint, so unless you have language metadata for your text, you can't render it correctly (which is half of was so bad about the pre-unicode days!). (Note: Unicode later added "variation selectors", which as far as I know solve the problems mentioned above. I don't know why the aren't in widespread use, or perhaps they are and I'm misunderstanding something.)

microcolonel 12y ago

I would like to take a moment to have people recognize that there are differences between the glyphs used for these characters, and some people find this to be a point of contention. For example, in the case of the text fragment “福祉”, there are three glyph representations for each character. Referenced below is an image demonstrating these three; first is Chinese, second is Korean, third is Japanese. [0] (I left out the Taiwanese etc. because they don't differ in this case from the CN) The UC have clear rules for this, as explained in their section "Characters, not glyphs", which starts on page 8 of the chapter 2 pdf of the Unicode 7.0 standard. They make the case that codepoints are not for representing stylistic or glyphic distinctions, as seen here, but rather for semantic separations; that is, people still agree that these are the same characters, even if they typeset and write them differently in Chinese, Korean, and Japanese text. They make cases specifically against a greco-latin unification, and for the CJK unification in Technical Note #26[1] [0] http://marumie.magnifi.ca:8080/ipfs/QmXDnJUPVtzxVtAQgFkNHwvkREL7DAKvgsLLEEWk9HAHNc/demonstration.svg http://marumie.magnifi.ca:8080/ipfs/QmXDnJUPVtzxVtAQgFkNHwvk... [1] http://www.unicode.org/notes/tn26/ http://www.unicode.org/notes/tn26/