4 ms·

Han unification was definitely a mistake. To this day people in different countries will use different fonts so that text looks how it is supposed to in their l

Han unification was definitely a mistake. To this day people in different countries will use different fonts so that text looks how it is supposed to in their language. The promise of unicode was that you can losslessly convert any encoding to unicode. However, because of the failed attempt at Han unification, some important information can be lost.

ksec 6y ago
Exactly, UTF-8 is great, except this part. In the name of unification you destroyed the culture that was there. But modern days people are fine with it as long as it is not their culture. And no, adding fonts or notation doesn't solve the problem. I remember I read a very nicely put analogy on HN years ago. Edit: https://news.ycombinator.com/item?id=8041288 https://news.ycombinator.com/item?id=8041288
- hnfong 6y ago
  The link describes the problems but _not_ having a unified CJK table results in other problems. I regularly read texts in both Simplified and Traditional Chinese and some Japanese, and it's nice that at least characters that are uncontroversially identical have the same code point, so that what I type in my preferred input method could be used interchangeably (think eg. text searching). There's enough trouble already with having a "handful" of commonly seen character variants between SC, TC and JP, making things like copyediting text a pain if you're not well versed in unicode shenanigans. I am involved in maintaining a Cantonese dictionary, and inadvertent duplicate entries occur very often because of these variants (yes, people sometimes, somehow, manage to input characters outside of the supposed locale -- and never notice because they look very similar). Fundamentally the problem is that European notions of "character" breaks down in East Asian languages. Not sure whether this is anyone's fault, and if we designed it again today with hindsight and improved (font rendering, text searching/processing etc.) technology we might have decided to construct Han "characters" with radicals instead. But in the 90s there might not have been a better choice. PS: not to mention the political fallout with having a separate table for Mainland-Chinese and Taiwan-Chinese if that were a policy. It's like having an English "a" and an American "a". Should Wales, Scotland and Northern Ireland have a different 'a' then? Why shouldn't the 50 states of USA each have a different 'a'. It'd bring the politics of the United Nations into Unicode.

ksec 6y ago

Exactly, UTF-8 is great, except this part. In the name of unification you destroyed the culture that was there. But modern days people are fine with it as long as it is not their culture. And no, adding fonts or notation doesn't solve the problem. I remember I read a very nicely put analogy on HN years ago. Edit: https://news.ycombinator.com/item?id=8041288 https://news.ycombinator.com/item?id=8041288

hnfong 6y ago
The link describes the problems but _not_ having a unified CJK table results in other problems. I regularly read texts in both Simplified and Traditional Chinese and some Japanese, and it's nice that at least characters that are uncontroversially identical have the same code point, so that what I type in my preferred input method could be used interchangeably (think eg. text searching). There's enough trouble already with having a "handful" of commonly seen character variants between SC, TC and JP, making things like copyediting text a pain if you're not well versed in unicode shenanigans. I am involved in maintaining a Cantonese dictionary, and inadvertent duplicate entries occur very often because of these variants (yes, people sometimes, somehow, manage to input characters outside of the supposed locale -- and never notice because they look very similar). Fundamentally the problem is that European notions of "character" breaks down in East Asian languages. Not sure whether this is anyone's fault, and if we designed it again today with hindsight and improved (font rendering, text searching/processing etc.) technology we might have decided to construct Han "characters" with radicals instead. But in the 90s there might not have been a better choice. PS: not to mention the political fallout with having a separate table for Mainland-Chinese and Taiwan-Chinese if that were a policy. It's like having an English "a" and an American "a". Should Wales, Scotland and Northern Ireland have a different 'a' then? Why shouldn't the 50 states of USA each have a different 'a'. It'd bring the politics of the United Nations into Unicode.