5 ms·

Could you elaborate?

by rjh29 3y ago

lmm 3y ago
Look up "Han Unification" - Unicode reuses codepoints for characters from Chinese and Japanese (also traditional Korean, but those characters aren't really used in modern Korean so it's much less of an issue), even though these characters actually look quite different. So any given font will look wrong in either China or Japan. Since China is the larger market, almost all "international" applications plump for a font that looks right in China and wrong in Japan.
- rjh29 3y ago
  Sure, I was wondering if there were any other issues. On my phone I just added Japanese as a language and it now prefers those. Windows seems to support it fine too, as does Chrome on Linux. I guess there are problems if you try to use two languages in the same application.
  - lmm 3y ago
    > On my phone I just added Japanese as a language and it now prefers those. Windows seems to support it fine too, as does Chrome on Linux. Is the text you're reading getting rendered in Japanese characters? Or is it getting written in similar-but-not-quite-the-same (and thus something you can use, but irritating to do so) Chinese characters?
    - rjh29 3y ago
      The former. You don't even need it as the primary language, secondary is enough. You can also set the language per app. (Samsung)
- DemocracyFTW2 3y ago
  As someone who has worked a lot with CJK in Unicode, I can not support the sentiment that Unicode uniquely screws Japanese. You'll always have to use an appropriate font for the task, the encoding won't help you. Yes, the idea of "Han Unification" was somewhat naive and even questionable, and one might ask whether it is still being applied to new codepoints, and there are some weirdities in CJK encoding (but those apply across all concerned languages and are not unique to Japanese). Then again, it is a difficult taks. For example, there are three ways to write the first (topmost) stroke of 言: as a slanted dot (now PRC standard), as a short vertical (very common in pre-war prints, still used in Japan) or as a short horizontal (also very common). Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. Although I've worked extensively on these problems—what is a variant, what deserves a codepoint—all I've found that there can and should be guidelines, but there's no very hard and fast rule. And I'm afraid variant selectors are a cop-out, not a solution.
  - lmm 3y ago
    > You'll always have to use an appropriate font for the task, the encoding won't help you. In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help. When you viewed a Japanese document it would be in a Japanese encoding and you'd get a Japanese font, when you viewed a Chinese document it would be in a Chinese encoding and you'd get a Chinese font, and when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. > but those apply across all concerned languages and are not unique to Japanese They're de facto unique to Japanese; Chinese support "won" in most international contexts, so "CJK" codepoints look fine in Chinese, and those codepoints are rarely used in Korean (mainly in historical documents) so having them rendered "wrong" there matters less in everyday life. Admittedly looking at another article that hit HN today I could believe Bulgaria gets equally shafted (and is probably, sadly, less capable of sustaining a "made in Bulgaria" software ecosystem than Japan is). > Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. You don't have to add 18 variants, only 3, and that's the worst case. We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant. But using Unicode shouldn't be a regression from using Shift-JIS.
    - DemocracyFTW2 3y ago
      > In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. > when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. > We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant Things are more complicated than that, I'm afraid. For one thing there's time, so setting the language in your HTML document or element to Japanese will give what is considered correct in Japanese today, not what was considered correct pre-war or any one time before that. There's no formal way to express that. Second, stylistic choices on one and language + region selector on the other are cross-cutting concerns. I have to say that I find your criticism not very well founded. What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) This will get you a long, long way to achieving correct and typographically pleasing output for any text be it Chinese traditional or simplified, Japanese, or Korean. It is some work, but then it is a vexing and somewhat convoluted problem, too. I think one should give font designers more credit here because good CJK fonts typically go to great lengths to satisfy even the finickier type heads among us, enabling variations that I'd frankly just gloss over for heavens sake. It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. So, no writing about Cuneiform or Hieroglyphs in Japanese. Today we can intermingle LTR and RTL scripts, and you can freely mix hundreds of scripts in a single document and a single encoding. None of that was possible in the good old days.

Could you elaborate?

by rjh29 3y ago

Could you elaborate?

lmm 3y ago

Look up "Han Unification" - Unicode reuses codepoints for characters from Chinese and Japanese (also traditional Korean, but those characters aren't really used in modern Korean so it's much less of an issue), even though these characters actually look quite different. So any given font will look wrong in either China or Japan. Since China is the larger market, almost all "international" applications plump for a font that looks right in China and wrong in Japan.

rjh29 3y ago
Sure, I was wondering if there were any other issues. On my phone I just added Japanese as a language and it now prefers those. Windows seems to support it fine too, as does Chrome on Linux. I guess there are problems if you try to use two languages in the same application.
- lmm 3y ago
  > On my phone I just added Japanese as a language and it now prefers those. Windows seems to support it fine too, as does Chrome on Linux. Is the text you're reading getting rendered in Japanese characters? Or is it getting written in similar-but-not-quite-the-same (and thus something you can use, but irritating to do so) Chinese characters?
  - rjh29 3y ago
    The former. You don't even need it as the primary language, secondary is enough. You can also set the language per app. (Samsung)
DemocracyFTW2 3y ago
As someone who has worked a lot with CJK in Unicode, I can not support the sentiment that Unicode uniquely screws Japanese. You'll always have to use an appropriate font for the task, the encoding won't help you. Yes, the idea of "Han Unification" was somewhat naive and even questionable, and one might ask whether it is still being applied to new codepoints, and there are some weirdities in CJK encoding (but those apply across all concerned languages and are not unique to Japanese). Then again, it is a difficult taks. For example, there are three ways to write the first (topmost) stroke of 言: as a slanted dot (now PRC standard), as a short vertical (very common in pre-war prints, still used in Japan) or as a short horizontal (also very common). Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. Although I've worked extensively on these problems—what is a variant, what deserves a codepoint—all I've found that there can and should be guidelines, but there's no very hard and fast rule. And I'm afraid variant selectors are a cop-out, not a solution.
- lmm 3y ago
  > You'll always have to use an appropriate font for the task, the encoding won't help you. In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help. When you viewed a Japanese document it would be in a Japanese encoding and you'd get a Japanese font, when you viewed a Chinese document it would be in a Chinese encoding and you'd get a Chinese font, and when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. > but those apply across all concerned languages and are not unique to Japanese They're de facto unique to Japanese; Chinese support "won" in most international contexts, so "CJK" codepoints look fine in Chinese, and those codepoints are rarely used in Korean (mainly in historical documents) so having them rendered "wrong" there matters less in everyday life. Admittedly looking at another article that hit HN today I could believe Bulgaria gets equally shafted (and is probably, sadly, less capable of sustaining a "made in Bulgaria" software ecosystem than Japan is). > Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. You don't have to add 18 variants, only 3, and that's the worst case. We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant. But using Unicode shouldn't be a regression from using Shift-JIS.
  - DemocracyFTW2 3y ago
    > In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. > when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. > We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant Things are more complicated than that, I'm afraid. For one thing there's time, so setting the language in your HTML document or element to Japanese will give what is considered correct in Japanese today, not what was considered correct pre-war or any one time before that. There's no formal way to express that. Second, stylistic choices on one and language + region selector on the other are cross-cutting concerns. I have to say that I find your criticism not very well founded. What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) This will get you a long, long way to achieving correct and typographically pleasing output for any text be it Chinese traditional or simplified, Japanese, or Korean. It is some work, but then it is a vexing and somewhat convoluted problem, too. I think one should give font designers more credit here because good CJK fonts typically go to great lengths to satisfy even the finickier type heads among us, enabling variations that I'd frankly just gloss over for heavens sake. It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. So, no writing about Cuneiform or Hieroglyphs in Japanese. Today we can intermingle LTR and RTL scripts, and you can freely mix hundreds of scripts in a single document and a single encoding. None of that was possible in the good old days.