4 ms·

As someone who has worked a lot with CJK in Unicode, I can not support the sentiment that Unicode uniquely screws Japanese. You'll always have to use an appropr

As someone who has worked a lot with CJK in Unicode, I can not support the sentiment that Unicode uniquely screws Japanese. You'll always have to use an appropriate font for the task, the encoding won't help you. Yes, the idea of "Han Unification" was somewhat naive and even questionable, and one might ask whether it is still being applied to new codepoints, and there are some weirdities in CJK encoding (but those apply across all concerned languages and are not unique to Japanese). Then again, it is a difficult taks. For example, there are three ways to write the first (topmost) stroke of 言: as a slanted dot (now PRC standard), as a short vertical (very common in pre-war prints, still used in Japan) or as a short horizontal (also very common). Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. Although I've worked extensively on these problems—what is a variant, what deserves a codepoint—all I've found that there can and should be guidelines, but there's no very hard and fast rule. And I'm afraid variant selectors are a cop-out, not a solution.

lmm 3y ago
> You'll always have to use an appropriate font for the task, the encoding won't help you. In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help. When you viewed a Japanese document it would be in a Japanese encoding and you'd get a Japanese font, when you viewed a Chinese document it would be in a Chinese encoding and you'd get a Chinese font, and when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. > but those apply across all concerned languages and are not unique to Japanese They're de facto unique to Japanese; Chinese support "won" in most international contexts, so "CJK" codepoints look fine in Chinese, and those codepoints are rarely used in Korean (mainly in historical documents) so having them rendered "wrong" there matters less in everyday life. Admittedly looking at another article that hit HN today I could believe Bulgaria gets equally shafted (and is probably, sadly, less capable of sustaining a "made in Bulgaria" software ecosystem than Japan is). > Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. You don't have to add 18 variants, only 3, and that's the worst case. We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant. But using Unicode shouldn't be a regression from using Shift-JIS.
- DemocracyFTW2 3y ago
  > In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. > when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. > We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant Things are more complicated than that, I'm afraid. For one thing there's time, so setting the language in your HTML document or element to Japanese will give what is considered correct in Japanese today, not what was considered correct pre-war or any one time before that. There's no formal way to express that. Second, stylistic choices on one and language + region selector on the other are cross-cutting concerns. I have to say that I find your criticism not very well founded. What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) This will get you a long, long way to achieving correct and typographically pleasing output for any text be it Chinese traditional or simplified, Japanese, or Korean. It is some work, but then it is a vexing and somewhat convoluted problem, too. I think one should give font designers more credit here because good CJK fonts typically go to great lengths to satisfy even the finickier type heads among us, enabling variations that I'd frankly just gloss over for heavens sake. It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. So, no writing about Cuneiform or Hieroglyphs in Japanese. Today we can intermingle LTR and RTL scripts, and you can freely mix hundreds of scripts in a single document and a single encoding. None of that was possible in the good old days.
  - lmm 3y ago
    > Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. Japan does (assuming agreeing a unicode-like encoding that actually worked is off the table), for the simple reason that practical support for Japanese in "international" applications has regressed since those days. > Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. https://wiki.c2.com/?AlmostCorrect https://wiki.c2.com/?AlmostCorrect . Failing obviously and universally is better than failing subtly in edge cases. > What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). > It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. The old system made the easy things easy. In those days effectively any program that could display French or Swedish documents could also display Japanese documents. The unicode world makes some things that were previously hard easier, such as mixing languages (as long as the lanugages you want to mix aren't Japanese), but your file format has to go beyond "plain" text for the most basic functionality of properly displaying monolingual Japanese documents. In the old days if you didn't think about languages at all your program would be broken in every non-English language. Now it's broken only in Japan. That's an improvement for most places, but it's a huge regression in Japan, because previously most programs made some effort to deal with internationalisation issues and now they don't, they just say "it's unicode so we don't have to think about it".
rjh29 3y ago
Well they did screw Japanese but gave Hong Kong/Taiwan the full set of traditional characters. Though that seems politically motivated. I wonder why they couldn't have done a composition system like Hangeul?
- DemocracyFTW2 3y ago
  > they did screw Japanese but gave Hong Kong/Taiwan the full set of traditional characters. Though that seems politically motivated. We don't live in a world where any part of this holds water, at least I don't. Care to elucidate where exactly HK and Taiwan got their full set but Japan didn't? > I wonder why they couldn't have done a composition system like Hangeul? The answer to that question is because it's too difficult. I worked a bit about automatic CJK character generation ion the early nineties, around the time Unicode v1 and then v2 came out. Nothing at the time resembled anything anywhere close to aesthetically pleasing output. Quite a number of people tried over decades, not a single paper demonstrates acceptable character shapes, only 'legible' and 'bearable' ones at most. Given recent advancements in AI text-to-image, it wouldn't surprise me if someone has already tried their hand at providing a model for CJK generation. But even then you'll be computationally much cheaper to just generate images for given code points plus those few unencoded characters that have to be generated ad-hoc.
  - lmm 3y ago
    > We don't live in a world where any part of this holds water, at least I don't. Care to elucidate where exactly HK and Taiwan got their full set but Japan didn't? Characters from "Traditional Chinese" and "Simplified Chinese" have different codepoints even for "the same" character, whereas characters from Japanese (or old Korean) are made to share Chinese characters' codepoints even for visually different characters.
    - DemocracyFTW2 3y ago
      The argument relies on 'visually different' being a well-defined, binary distinction. It is not. 'Similarity' and 'dissimilarity' can at best be described for a narrow set of purposes with guiding rules. So is the exact shape of the vertical or vertically-slanted dot on top of 言 enough to trigger the dissimilarity warning, or is it only when it's written as a horizontal? How about 吉田 vs. ⿱土口田? Is that similar or dissimilar? How about that little 'hair' stroke (ヒゲ or 筆押さえ) that is sometimes seen in characters like 文? When you compare the samples shown over at http://tonan.seesaa.net/article/431481813.html http://tonan.seesaa.net/article/431481813.html, wouldn't you agree that it is quite difficult to predict which characters should and which ones shouldn't have a ヒゲ? Did you know that the JIS 1983 edition did include the ヒゲ in many characters where the JIS 1990 edition omitted it? You'll find all the gory details of what changed between JIS encoding editions in this fine book: 大熊肇: 文字の骨組[1], esp. pp164—178 where all the fine details between different editions of the JIS encoding are listed. It's mind-boggling! It's almost as though the Japanese standards body itself has been having slightly different opinions about their own writing system over the decades. Now if you're intent on getting all the fine details just right for your print edition you can either apply a newer or older JIS encoding to your document, OR invent that file format you're talking about where you can switch encodings mid-way, OR access OpenType font features from your document somehow... mmmh... how to do that... mmmh... Oh I know! Let's use Unicode and HTML and CSS and get full access to OpenType font features for any point in the document on a per-codepoint basis. For those folks who really pine for the bad old days when all we had was US ASCII plus an insane number of other encodings, please do read this single Wikipedia article: https://en.wikipedia.org/wiki/ISO/IEC_2022 https://en.wikipedia.org/wiki/ISO/IEC_2022 it will change your mind. * [1] https://www.amazon.co.jp/%E6%96%87%E5%AD%97%E3%81%AE%E9%AA%A8%E7%B5%84%E3%81%BF%E2%80%95%E5%AD%97%E4%BD%93-%E7%94%B2%E9%AA%A8%E6%96%87%E3%81%8B%E3%82%89%E5%B8%B8%E7%94%A8%E6%BC%A2%E5%AD%97%E3%81%BE%E3%81%A7-%E5%A4%A7%E7%86%8A-%E8%82%87/dp/4434130919 https://www.amazon.co.jp/%E6%96%87%E5%AD%97%E3%81%AE%E9%AA%A...

lmm 3y ago

> You'll always have to use an appropriate font for the task, the encoding won't help you. In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help. When you viewed a Japanese document it would be in a Japanese encoding and you'd get a Japanese font, when you viewed a Chinese document it would be in a Chinese encoding and you'd get a Chinese font, and when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. > but those apply across all concerned languages and are not unique to Japanese They're de facto unique to Japanese; Chinese support "won" in most international contexts, so "CJK" codepoints look fine in Chinese, and those codepoints are rarely used in Korean (mainly in historical documents) so having them rendered "wrong" there matters less in everyday life. Admittedly looking at another article that hit HN today I could believe Bulgaria gets equally shafted (and is probably, sadly, less capable of sustaining a "made in Bulgaria" software ecosystem than Japan is). > Sure, one can make three codepoints out of these, no big deal. But then 言 occurs in a gazillion of other characters, as in 說這語信 and so on; all of these characters would then need to get codepoints of their own (which btw is exactly what happened with 說 vs 説 (and incidentally 说)). It gets worse when you consider that another one or two components of a given character may also have two or three variants, then you might get 3x2=6 or 3x2x3=18 codepoints for what most people consider a single character. And did I mention we already have around 100'000 codepoints for CJK Ideographs? That number would multiply considerably. You don't have to add 18 variants, only 3, and that's the worst case. We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant. But using Unicode shouldn't be a regression from using Shift-JIS.

DemocracyFTW2 3y ago
> In practice in the pre-unicode world fonts were encoding-specific, so the encoding did help Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. > when you mixed up your encodings you'd get something that was obviously broken even to someone doesn't speak Chinese or Japanese. Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. > We don't need codepoints for letters that are half-Japanese half-Korean, or every other conceivable variant Things are more complicated than that, I'm afraid. For one thing there's time, so setting the language in your HTML document or element to Japanese will give what is considered correct in Japanese today, not what was considered correct pre-war or any one time before that. There's no formal way to express that. Second, stylistic choices on one and language + region selector on the other are cross-cutting concerns. I have to say that I find your criticism not very well founded. What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) This will get you a long, long way to achieving correct and typographically pleasing output for any text be it Chinese traditional or simplified, Japanese, or Korean. It is some work, but then it is a vexing and somewhat convoluted problem, too. I think one should give font designers more credit here because good CJK fonts typically go to great lengths to satisfy even the finickier type heads among us, enabling variations that I'd frankly just gloss over for heavens sake. It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. So, no writing about Cuneiform or Hieroglyphs in Japanese. Today we can intermingle LTR and RTL scripts, and you can freely mix hundreds of scripts in a single document and a single encoding. None of that was possible in the good old days.
- lmm 3y ago
  > Nobody wants to go back to a world with hundreds of relevant and thousands of legacy encodings. Japan does (assuming agreeing a unicode-like encoding that actually worked is off the table), for the simple reason that practical support for Japanese in "international" applications has regressed since those days. > Obviously much better when nobody can read anything than forcing some people to read texts where there's the occasional odd ('non-native' if you exaggerate a bit) font style choice. https://wiki.c2.com/?AlmostCorrect https://wiki.c2.com/?AlmostCorrect . Failing obviously and universally is better than failing subtly in edge cases. > What you can do today is (taking HTML+CSS as the obvious choice) is proper font choice, `@font-face` declarations, proper downloadable fonts, proper configuration of OpenType font feature flags, if necessary on the basis of Unicode code point ranges. (There's also Unicode variation selectors, bit I've never worked with those so can't say how well those work.) You can do a lot in HTML because HTML has a lot of specific support - something that would likely also be true in a world without unicode. (In the unicode world we have only one encoding per HTML file, but there's no reason a file format couldn't support multiple encodings in the same way as supporting multiple language spans etc.). > It's not like the pre-Unicode chaos of mutually incompatible encodings did anything to help you with any of these points. Rather, it locked you into a rather small (when compared to Unicode) set of codepoints, and you typically only could do one setting per document. The old system made the easy things easy. In those days effectively any program that could display French or Swedish documents could also display Japanese documents. The unicode world makes some things that were previously hard easier, such as mixing languages (as long as the lanugages you want to mix aren't Japanese), but your file format has to go beyond "plain" text for the most basic functionality of properly displaying monolingual Japanese documents. In the old days if you didn't think about languages at all your program would be broken in every non-English language. Now it's broken only in Japan. That's an improvement for most places, but it's a huge regression in Japan, because previously most programs made some effort to deal with internationalisation issues and now they don't, they just say "it's unicode so we don't have to think about it".

rjh29 3y ago

Well they did screw Japanese but gave Hong Kong/Taiwan the full set of traditional characters. Though that seems politically motivated. I wonder why they couldn't have done a composition system like Hangeul?

DemocracyFTW2 3y ago
> they did screw Japanese but gave Hong Kong/Taiwan the full set of traditional characters. Though that seems politically motivated. We don't live in a world where any part of this holds water, at least I don't. Care to elucidate where exactly HK and Taiwan got their full set but Japan didn't? > I wonder why they couldn't have done a composition system like Hangeul? The answer to that question is because it's too difficult. I worked a bit about automatic CJK character generation ion the early nineties, around the time Unicode v1 and then v2 came out. Nothing at the time resembled anything anywhere close to aesthetically pleasing output. Quite a number of people tried over decades, not a single paper demonstrates acceptable character shapes, only 'legible' and 'bearable' ones at most. Given recent advancements in AI text-to-image, it wouldn't surprise me if someone has already tried their hand at providing a model for CJK generation. But even then you'll be computationally much cheaper to just generate images for given code points plus those few unencoded characters that have to be generated ad-hoc.
- lmm 3y ago
  > We don't live in a world where any part of this holds water, at least I don't. Care to elucidate where exactly HK and Taiwan got their full set but Japan didn't? Characters from "Traditional Chinese" and "Simplified Chinese" have different codepoints even for "the same" character, whereas characters from Japanese (or old Korean) are made to share Chinese characters' codepoints even for visually different characters.
  - DemocracyFTW2 3y ago
    The argument relies on 'visually different' being a well-defined, binary distinction. It is not. 'Similarity' and 'dissimilarity' can at best be described for a narrow set of purposes with guiding rules. So is the exact shape of the vertical or vertically-slanted dot on top of 言 enough to trigger the dissimilarity warning, or is it only when it's written as a horizontal? How about 吉田 vs. ⿱土口田? Is that similar or dissimilar? How about that little 'hair' stroke (ヒゲ or 筆押さえ) that is sometimes seen in characters like 文? When you compare the samples shown over at http://tonan.seesaa.net/article/431481813.html http://tonan.seesaa.net/article/431481813.html, wouldn't you agree that it is quite difficult to predict which characters should and which ones shouldn't have a ヒゲ? Did you know that the JIS 1983 edition did include the ヒゲ in many characters where the JIS 1990 edition omitted it? You'll find all the gory details of what changed between JIS encoding editions in this fine book: 大熊肇: 文字の骨組[1], esp. pp164—178 where all the fine details between different editions of the JIS encoding are listed. It's mind-boggling! It's almost as though the Japanese standards body itself has been having slightly different opinions about their own writing system over the decades. Now if you're intent on getting all the fine details just right for your print edition you can either apply a newer or older JIS encoding to your document, OR invent that file format you're talking about where you can switch encodings mid-way, OR access OpenType font features from your document somehow... mmmh... how to do that... mmmh... Oh I know! Let's use Unicode and HTML and CSS and get full access to OpenType font features for any point in the document on a per-codepoint basis. For those folks who really pine for the bad old days when all we had was US ASCII plus an insane number of other encodings, please do read this single Wikipedia article: https://en.wikipedia.org/wiki/ISO/IEC_2022 https://en.wikipedia.org/wiki/ISO/IEC_2022 it will change your mind. * [1] https://www.amazon.co.jp/%E6%96%87%E5%AD%97%E3%81%AE%E9%AA%A8%E7%B5%84%E3%81%BF%E2%80%95%E5%AD%97%E4%BD%93-%E7%94%B2%E9%AA%A8%E6%96%87%E3%81%8B%E3%82%89%E5%B8%B8%E7%94%A8%E6%BC%A2%E5%AD%97%E3%81%BE%E3%81%A7-%E5%A4%A7%E7%86%8A-%E8%82%87/dp/4434130919 https://www.amazon.co.jp/%E6%96%87%E5%AD%97%E3%81%AE%E9%AA%A...