4 ms·

The parent said "displayed correctly", not "encoded". For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your

The parent said "displayed correctly", not "encoded". For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer. The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character. Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese. See https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/

int_19h 3y ago
How does Shift-JIS fix this?
- lmm 3y ago
  The Shift-JIS codepoints for the characters of that name are understood to refer to Japanese characters, so fonts render them correctly. Encoding-aware programs have different representations (such as GB 18030 codepoints) for the similar-but-different Chinese characters that unicode-only programs tend to display these characters as, and so will render them differently.
  - int_19h 3y ago
    If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
    - lmm 2y ago
      > If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. Only if you have some unicodebrained mentality where you consider a Chinese character that looks sort of similar to a Japanese character to be "the same". If you think of them as two different characters then they're just different characters, which may or may not be present in particular encodings (which is completely normal if you make a program that handles multiple encodings: not every character exists in every encoding) > In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters? Firstly, you have to "draw the rest of the fucking owl" and actually implement that language selection mechanism. Secondly, if you implement some clever extension mechanism on top of UTF-8 that's only needed for Japanese and can only really be tested by people who read Chinese or Japanese, realistically even if you implement it perfectly, most app makers won't use it or will use it wrong. Whereas if you implement encoding-awareness in the standard way that we've been doing for decades and test with even one non-default encoding, your application will most likely work fine for Japanese, Chinese and every other language even if you never test it with Japanese.
- TheDong 3y ago
  I didn't propose any solution or claim Shift-JIS fixes this. It doesn't really since a single Shift-JIS document can only encode the japanese variant, not both. However, a unicode codepoint which acted as a "language hint" switch would solve this, and wouldn't require doubling the number of han codepoints. There already are unicode variant selectors, but iiuc they only apply to a single character at a time, and no one actually uses them, so they're not very useful.

int_19h 3y ago

How does Shift-JIS fix this?

lmm 3y ago
The Shift-JIS codepoints for the characters of that name are understood to refer to Japanese characters, so fonts render them correctly. Encoding-aware programs have different representations (such as GB 18030 codepoints) for the similar-but-different Chinese characters that unicode-only programs tend to display these characters as, and so will render them differently.
- int_19h 3y ago
  If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
  - lmm 2y ago
    > If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. Only if you have some unicodebrained mentality where you consider a Chinese character that looks sort of similar to a Japanese character to be "the same". If you think of them as two different characters then they're just different characters, which may or may not be present in particular encodings (which is completely normal if you make a program that handles multiple encodings: not every character exists in every encoding) > In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters? Firstly, you have to "draw the rest of the fucking owl" and actually implement that language selection mechanism. Secondly, if you implement some clever extension mechanism on top of UTF-8 that's only needed for Japanese and can only really be tested by people who read Chinese or Japanese, realistically even if you implement it perfectly, most app makers won't use it or will use it wrong. Whereas if you implement encoding-awareness in the standard way that we've been doing for decades and test with even one non-default encoding, your application will most likely work fine for Japanese, Chinese and every other language even if you never test it with Japanese.
TheDong 3y ago
I didn't propose any solution or claim Shift-JIS fixes this. It doesn't really since a single Shift-JIS document can only encode the japanese variant, not both. However, a unicode codepoint which acted as a "language hint" switch would solve this, and wouldn't require doubling the number of han codepoints. There already are unicode variant selectors, but iiuc they only apply to a single character at a time, and no one actually uses them, so they're not very useful.