5 ms·

> China and Japan could have stayed with their own encodings. Absolutely correct. There was a big debate in Japan in the 1990s about character encodings, with

> China and Japan could have stayed with their own encodings. Absolutely correct. There was a big debate in Japan in the 1990s about character encodings, with some people arguing strongly against the adoption of Unicode. Their main argument, as I remember it, was that Unicode didn’t capture all of the variations in kanji, especially for personal names. For those of us who were trying to use Japanese online at the time, though, those arguments seemed beside the point. While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1], we were faced with the daily frustration of trying to convert between JIS, S-JIS, EUC, and other encodings and often not being able to exchange Japanese text at all with people who hadn’t installed special software on their computers. It was a great relief when UTF-8 became adopted universally. And now we have emoji, too! [1] https://www.fujitv-view.jp/gallery/post-149246/?imgid=1 https://www.fujitv-view.jp/gallery/post-149246/?imgid=1

ant6n 6y ago
> While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1],... Unicode has gotten so big, isn’t this included by now?
- bmn__ 6y ago
  http://enwp.org/Z-variant http://enwp.org/Z-variant
  - lifthrasiir 6y ago
    Also see the IVD [1]. Indeed both 邉 (U+9089) and 邊 (U+908A) are exceptionally variable characters, the first having 32 variation sequences (the record as of 2020-11-06) and the second having 21 variation sequences. [1] https://unicode.org/ivd/ https://unicode.org/ivd/
anthk 6y ago
Does the Japanese goverment use Hiragana on official documments in order to properly spell out names easily?
- tkgally 6y ago
  Either hiragana or katakana is used on most official documents for convenience. On the family registers (戸籍 koseki), which are perhaps the most important, though, the readings of names are not listed. For people whose names are written only in kanji, those kanji, and not the readings, are the legal versions of their names.
pezezin 6y ago
Tell that to my coworkers. I still get emails encoded in SJIS every day, sometimes with attachments with the file name also encoded in SJIS, which results in funny mojibake when saving them to disk. Not to mention the many web forms that insist you need to write your name in full-width characters or whatever funky shit. On the other I recently got some Python scripts to crash because someone in the European team decided to encode some texts in ISO-8859-1 and Python assumes everything is in UTF-8. I really, really wish one day all legacy encodings will disappear from the face of the Earth and only UTF-8 will stay.
- anthk 6y ago
  Iconv is your friend. On that issue, infsp6 (the Spanish library for Inform6, akin to the English inform6lib one) still uses iso8859-15 and it's a pain in the ass to convert the encoding to and from utf8 if you don't use neither emacs, joe or vim to edit the source code (I use nvi).
  - pezezin 6y ago
    Mátalos a todos y que dios elija a los suyos. But at least it's not EBCDIC, the day I find that in the wild is the day I will retire from computers and become a farmer.
- aYsY4dDQ2NrcNzA 6y ago
  Out of curiosity, have you tried the UTF-8 decoder capability and stress test? https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html
  - pezezin 6y ago
    No, why? Thunderbird will display SJIS emails just fine. The problem with attachments is when some adds a ZIP with SJIS filenames, but then it's not Thunderbird's problem but whatever tool you use to decompress it. Regarding Python, the default behaviour when decoding and invalid UTF-8 strings is to raise an exception. But your comment made me research it and I just found that there is a way to replace invalid bytes with U+FFFD, so I will try it.
lmm 6y ago
SJIS is still pretty actively used, and Han unification is the most likely culprit. In hindsight it really does feel like a mistake.
- ChrisSD 6y ago
  Han unification was definitely a mistake. To this day people in different countries will use different fonts so that text looks how it is supposed to in their language. The promise of unicode was that you can losslessly convert any encoding to unicode. However, because of the failed attempt at Han unification, some important information can be lost.
  - ksec 6y ago
    Exactly, UTF-8 is great, except this part. In the name of unification you destroyed the culture that was there. But modern days people are fine with it as long as it is not their culture. And no, adding fonts or notation doesn't solve the problem. I remember I read a very nicely put analogy on HN years ago. Edit: https://news.ycombinator.com/item?id=8041288 https://news.ycombinator.com/item?id=8041288
    - hnfong 6y ago
      The link describes the problems but _not_ having a unified CJK table results in other problems. I regularly read texts in both Simplified and Traditional Chinese and some Japanese, and it's nice that at least characters that are uncontroversially identical have the same code point, so that what I type in my preferred input method could be used interchangeably (think eg. text searching). There's enough trouble already with having a "handful" of commonly seen character variants between SC, TC and JP, making things like copyediting text a pain if you're not well versed in unicode shenanigans. I am involved in maintaining a Cantonese dictionary, and inadvertent duplicate entries occur very often because of these variants (yes, people sometimes, somehow, manage to input characters outside of the supposed locale -- and never notice because they look very similar). Fundamentally the problem is that European notions of "character" breaks down in East Asian languages. Not sure whether this is anyone's fault, and if we designed it again today with hindsight and improved (font rendering, text searching/processing etc.) technology we might have decided to construct Han "characters" with radicals instead. But in the 90s there might not have been a better choice. PS: not to mention the political fallout with having a separate table for Mainland-Chinese and Taiwan-Chinese if that were a policy. It's like having an English "a" and an American "a". Should Wales, Scotland and Northern Ireland have a different 'a' then? Why shouldn't the 50 states of USA each have a different 'a'. It'd bring the politics of the United Nations into Unicode.

ant6n 6y ago

> While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1],... Unicode has gotten so big, isn’t this included by now?

bmn__ 6y ago
http://enwp.org/Z-variant http://enwp.org/Z-variant
- lifthrasiir 6y ago
  Also see the IVD [1]. Indeed both 邉 (U+9089) and 邊 (U+908A) are exceptionally variable characters, the first having 32 variation sequences (the record as of 2020-11-06) and the second having 21 variation sequences. [1] https://unicode.org/ivd/ https://unicode.org/ivd/

anthk 6y ago

Does the Japanese goverment use Hiragana on official documments in order to properly spell out names easily?

tkgally 6y ago
Either hiragana or katakana is used on most official documents for convenience. On the family registers (戸籍 koseki), which are perhaps the most important, though, the readings of names are not listed. For people whose names are written only in kanji, those kanji, and not the readings, are the legal versions of their names.

pezezin 6y ago

Tell that to my coworkers. I still get emails encoded in SJIS every day, sometimes with attachments with the file name also encoded in SJIS, which results in funny mojibake when saving them to disk. Not to mention the many web forms that insist you need to write your name in full-width characters or whatever funky shit. On the other I recently got some Python scripts to crash because someone in the European team decided to encode some texts in ISO-8859-1 and Python assumes everything is in UTF-8. I really, really wish one day all legacy encodings will disappear from the face of the Earth and only UTF-8 will stay.

anthk 6y ago
Iconv is your friend. On that issue, infsp6 (the Spanish library for Inform6, akin to the English inform6lib one) still uses iso8859-15 and it's a pain in the ass to convert the encoding to and from utf8 if you don't use neither emacs, joe or vim to edit the source code (I use nvi).
- pezezin 6y ago
  Mátalos a todos y que dios elija a los suyos. But at least it's not EBCDIC, the day I find that in the wild is the day I will retire from computers and become a farmer.
aYsY4dDQ2NrcNzA 6y ago
Out of curiosity, have you tried the UTF-8 decoder capability and stress test? https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html
- pezezin 6y ago
  No, why? Thunderbird will display SJIS emails just fine. The problem with attachments is when some adds a ZIP with SJIS filenames, but then it's not Thunderbird's problem but whatever tool you use to decompress it. Regarding Python, the default behaviour when decoding and invalid UTF-8 strings is to raise an exception. But your comment made me research it and I just found that there is a way to replace invalid bytes with U+FFFD, so I will try it.

lmm 6y ago

SJIS is still pretty actively used, and Han unification is the most likely culprit. In hindsight it really does feel like a mistake.

ChrisSD 6y ago
Han unification was definitely a mistake. To this day people in different countries will use different fonts so that text looks how it is supposed to in their language. The promise of unicode was that you can losslessly convert any encoding to unicode. However, because of the failed attempt at Han unification, some important information can be lost.
- ksec 6y ago
  Exactly, UTF-8 is great, except this part. In the name of unification you destroyed the culture that was there. But modern days people are fine with it as long as it is not their culture. And no, adding fonts or notation doesn't solve the problem. I remember I read a very nicely put analogy on HN years ago. Edit: https://news.ycombinator.com/item?id=8041288 https://news.ycombinator.com/item?id=8041288
  - hnfong 6y ago
    The link describes the problems but _not_ having a unified CJK table results in other problems. I regularly read texts in both Simplified and Traditional Chinese and some Japanese, and it's nice that at least characters that are uncontroversially identical have the same code point, so that what I type in my preferred input method could be used interchangeably (think eg. text searching). There's enough trouble already with having a "handful" of commonly seen character variants between SC, TC and JP, making things like copyediting text a pain if you're not well versed in unicode shenanigans. I am involved in maintaining a Cantonese dictionary, and inadvertent duplicate entries occur very often because of these variants (yes, people sometimes, somehow, manage to input characters outside of the supposed locale -- and never notice because they look very similar). Fundamentally the problem is that European notions of "character" breaks down in East Asian languages. Not sure whether this is anyone's fault, and if we designed it again today with hindsight and improved (font rendering, text searching/processing etc.) technology we might have decided to construct Han "characters" with radicals instead. But in the 90s there might not have been a better choice. PS: not to mention the political fallout with having a separate table for Mainland-Chinese and Taiwan-Chinese if that were a policy. It's like having an English "a" and an American "a". Should Wales, Scotland and Northern Ireland have a different 'a' then? Why shouldn't the 50 states of USA each have a different 'a'. It'd bring the politics of the United Nations into Unicode.