3 ms·
SJIS is still pretty actively used, and Han unification is the most likely culprit. In hindsight it really does feel like a mistake.
by lmm 6y ago
SJIS is still pretty actively used, and Han unification is the most likely culprit. In hindsight it really does feel like a mistake.
- ChrisSD 6y agoHan unification was definitely a mistake. To this day people in different countries will use different fonts so that text looks how it is supposed to in their language. The promise of unicode was that you can losslessly convert any encoding to unicode. However, because of the failed attempt at Han unification, some important information can be lost.
- ksec 6y agoExactly, UTF-8 is great, except this part. In the name of unification you destroyed the culture that was there. But modern days people are fine with it as long as it is not their culture. And no, adding fonts or notation doesn't solve the problem. I remember I read a very nicely put analogy on HN years ago. Edit: https://news.ycombinator.com/item?id=8041288 https://news.ycombinator.com/item?id=8041288
- hnfong 6y agoThe link describes the problems but _not_ having a unified CJK table results in other problems. I regularly read texts in both Simplified and Traditional Chinese and some Japanese, and it's nice that at least characters that are uncontroversially identical have the same code point, so that what I type in my preferred input method could be used interchangeably (think eg. text searching). There's enough trouble already with having a "handful" of commonly seen character variants between SC, TC and JP, making things like copyediting text a pain if you're not well versed in unicode shenanigans. I am involved in maintaining a Cantonese dictionary, and inadvertent duplicate entries occur very often because of these variants (yes, people sometimes, somehow, manage to input characters outside of the supposed locale -- and never notice because they look very similar). Fundamentally the problem is that European notions of "character" breaks down in East Asian languages. Not sure whether this is anyone's fault, and if we designed it again today with hindsight and improved (font rendering, text searching/processing etc.) technology we might have decided to construct Han "characters" with radicals instead. But in the 90s there might not have been a better choice. PS: not to mention the political fallout with having a separate table for Mainland-Chinese and Taiwan-Chinese if that were a policy. It's like having an English "a" and an American "a". Should Wales, Scotland and Northern Ireland have a different 'a' then? Why shouldn't the 50 states of USA each have a different 'a'. It'd bring the politics of the United Nations into Unicode.
- earthboundkid 6y agoHan unification is a bullshit excuse. Is two story 'a' a different letter than one story? Is seven with a slash through it different than seven without? Is Japanese as written in pre-war books a different language than Japanese in post-war books? Unicode may have dropped a couple of variants, but they basically all got added back. There's no problem with Han unification; there's just a FUD campaign powered by nationalism and ignorance that is used to justify everyday technological inertia.
- fomine3 6y agoHan unification isn't main blocker to transition to utf-8 in Japan. Just use Japanese font in this context. Why SJIS is still used is because there are many legacy systems and developers who still think SJIS is fine. We not tend to treat other languages so it mostly works (without emoji). SJIS is sometimes useful because 1-byte char is half width, and 2-bytes char is full width by design. Old developers still call Japanese character as "2-bytes character" even though the system is utf-8. Another reason is that Windows OEM codepage is still CP932 (extended SJIS). It's pain like this: https://discuss.python.org/t/pep-597-enable-utf-8-mode-by-default-on-windows/3122 https://discuss.python.org/t/pep-597-enable-utf-8-mode-by-de...