4 ms·

I'm a native-CJK user myself and well aware this phenomenon, but honestly it's not really that bad in most of cases due to the following reasons: 1. Websites t

I'm a native-CJK user myself and well aware this phenomenon, but honestly it's not really that bad in most of cases due to the following reasons: 1. Websites that are in Japanese are likely tagged with lang=ja already. So they will display fine. Unfortunately, this practice seems to be less followed by Chinese sites. I checked a few top sites, qq.com do have lang=zh-cn, while baidu.com and sina.com.cn don't. 2. Majority of UI elements in OS will prioritize the display language you set when choosing variants. This means, if the users are reading content in Japanese while also using Japanese UI, the glyphs would be correct. Of course, this will cause problem if a Japanese is reading Chinese or vice versa, but such scenario is in minority. Another scenario, which I think is more common, is when someone is using a Latin-language UI. For example, lots of my (Chinese/Japanese) friends are using English UI while reading Chinese/Japanese a lot. The OS in this case will default to one variant (I believe Apple by default would choose Japanese) and therefore display another language's glyphs wrong (side note: for web pages, desktop browsers often have their own font/glyph fallback logic above the OS one). 3. Most of people are just not sensitive to such thing. I pointed it out to lots of people (when due to their setting, some glyphs are displayed wrong, like 门), and they can't care less. Also, there is no simple "fix" if you have multi-language content. Without manually assign <lang> tag to every single string, you can't display both Japanese and Chinese correct at the same time. It isn't worth the hassle for just a few phrases in text. A good example is Wikipedia, they have templates for all kinds of languages so you can display them correctly even if it's just one Japanese word on, say, English Wikipedia. And Wiki editors do use them all the time!

eloisant 5y ago
What you're saying is true (although I'm not sure about 3, all the Japanese people I've talked to are annoyed by that), but it really sucks that we're dealing with problems that were supposed to be fixed by Unicode. Han unification have been a huge mistake, to save a few thousands of characters, and now we keep piling on more and more stupid emoji.
- dotancohen 5y ago
  > Han unification have been a huge mistake, to save a few thousands > of characters, and now we keep piling on more and more stupid emoji. This is my exact problem with Unicode. I've very grateful for the efforts that they have made in the past, but the change from "spare valuable codepoints at the expense of causing ambiguity in text" to "assign a new codepoint to every cartoon permutation of intangible nouns" is infuriating.
  - oleganza 5y ago
    25 years ago computers and networks were different. Today text is a <0.1% of traffic compared with video and you have billions of bytes of RAM in every pocket computer. So yes, you can pile on more emojis and no one would be bothered. Unicode may have been never adopted at all if it had even larger set for CJK and made all CJK texts 1.5-2x larger than in Han-unified version, due to longer encoding. Also: UTF-8 did not exist yet and most systems treated text as arrays of fixed length characters.
    - anthk 5y ago
      UTF-8 existed 25 years ago, but in Plan9.
  - arp242 5y ago
    Han unification and Emojis are from a different time. We're talking about a decision made about 30 years ago in the early 90s; Unicode was 16 bits (65k total codepoints) at the time. The original CJK Unified Ideographs block from 1992 consists of 21k codepoints. It was impossible to do that four times since 4×21k is more than 65k, and we're going to need space for some other languages as well. Why not make Unicode larger? Well, size was a real concern back then (still is, to some degree, but less so). Since then Unicode has expanded and now we have slightly under 1 million codepoints. Han character blocks have extended to about 93k codepoints today, and 4 times ~93k codepoints is actually feasible. But now you run in to compatibility issues: you can't remove all the old Han unification stuff (it will break text, big no-no), so you need to re-define it all anew. Is that better? How about mixing "old" Han unified codepoints with new Japanese or Chinese stuff? Will it really improve things or just cause endless confusion (see: combining characters)? For scale, all of Unicode currently defines about 145k codepoints; so even with Han unification we're talking about two thirds being taken up by just these three languages. In comparison there are currently about 3,000 emojis, although the number of codepoints is much less since many codepoints are re-used (e.g. "firefighter" is "person + firetruck", flags use the country code, etc.). In a quick check it looks like there are about 1,000 to 1,500 codepoints reserved for emojis. In comparison, this is nothing. What I'm trying to say is that the (comparatively) very low number of emojis has absolutely no bearing on this and that going off on a tangent about it is very misplaced.
gpderetta 5y ago
Is there a reason unicode doesn't have such a builtin lang tag? Similar to right-to-left and left-to-right it could help in displaying differently otherwise identical text. It could be stored in-band with the text with little changes to existing systems. The only change would be on the presentation layer, and if the tag were to be a non printable character, it would be backward compatible. An input device could implicitly tag input texts depending on the default lang. You need some form of sanitization, but you need it for right-to-left and left-to-right already.
- arp242 5y ago
  Actually, this exists already – there's U+E0001 (LANGUAGE TAG) and U+E007 (CANCEL TAG) and you can put a language code between those, e.g. "\uE0001ja-JP\ue007". Its use is also deprecated and discouraged. According to [1] it's often not needed, and [2] states that it puts a lot of burden on implementations and best done at a higher level such as HTTP, HTML, etc. I have no opinion on [1] as I don't speak these languages, but I do know I really hate working with these "invisible characters" in Unicode both as a user and developer. Copy an extra invisible LTR thingy or display variant codepoint and stuff can look and behave different, and it may not at all be obvious what the hell is going on (especially for those without a technical background). [1]: https://www.unicode.org/versions/Unicode14.0.0/ch05.pdf#G11552 https://www.unicode.org/versions/Unicode14.0.0/ch05.pdf#G115... [2]: https://www.unicode.org/versions/Unicode14.0.0/ch23.pdf#G30110 https://www.unicode.org/versions/Unicode14.0.0/ch23.pdf#G301...
  - lifthrasiir 5y ago
    > there's U+E0001 (LANGUAGE TAG) and U+E007 (CANCEL TAG) and you can put a language code between those, e.g. "\uE0001ja-JP\ue007". "ja-JP" part is also written in tag characters, so it's actually E0001 E006A E0061 E002D E004A E0050 E007F and doesn't render even in unsupported environments.
    - arp242 5y ago
      Oh right; I never actually used it, I just know it exists.
marginalia_nu 5y ago
Re: lang="jp" My experience with web crawling is that the use of the lang-tag seems inconsistent at best. To make matters worse, sometimes content is straight up mislabeled, although Japanese sites often helpfully declare that they are using the Shift_JIS charset rather than UTF-8, which is at least somewhat helpful in figuring out that it is Japanese.
cehrlich 5y ago
I agree that for CJK-natives it's not such a big deal probably (unless they live their life in more than one of those languages). For people like me who primarily use their computer in English but also do some stuff in Japanese every now and then it's very frustrating. Of course at this point I know what's going on when 直 or whatever looks wrong, but it's still frustrating. OSs and Browsers having their own logic for it actually makes things _worse_ in some cases. Windows is especially bad (different types of UI elements care about different settings or don't care at all, so good luck having apps render correctly if you don't change your entire OS locale), and Chrome is pretty bad too, again especially on Windows. Overall MacOS/iOS and Safari does the best job by far. The failed attempt at Han Unification[1] is the worst decision the Unicode people have ever made. [1] https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
- jandrese 5y ago
  I disagree. UCS-2 was the worst decision the Unicode consortium ever made. Han unification is merely a side effect of this bad decision.
- nanis 5y ago
  > The failed attempt at Han Unification[1] is the worst decision the Unicode people have ever made. At first I nodded my head in agreement, but then I decided I still think the failure to include separate code points for "lower case Turkish dotted I" and "upper case Turkish dotless I" is worse. You can't have 'ı' ≡ lc( uc 'ı' ) unless you already know you are processing Turkish ... completely unnecessary complication.
  - naniwaduni 5y ago
    Turkish I unification, at least, wasn't a decision the Unicode people made, they inherited the mistake from earlier encodings. Given that those already existed, the alternative to having broken casefolding was, essentially, break all mixed Turkish documents transcoded from cp857 containing both "I" and "i" in non-Turkish functional directives, i.e. you'd necessarily break things like HTML documents without consistent tag casing.
needle0 5y ago
My original motive to write this page came most from issues in video games, where text is often displayed using custom routines and built-in rules in the OS & browser can't be of help. The issue crops up most often in indie games, but they can be seen even in high-profile high-budget games like Half-Life: Alyx or Resident Evil 4 VR.
makeitdouble 5y ago
> 3. Most of people are just not sensitive to such thing. I pointed it out to lots of people (when due to their setting, some glyphs are displayed wrong, like 门), and they can't care less. I think its' because most people don't deal with it in big amounts. I heard a lot more complaints from people using android phones that didn't have jp fonts by default. At the third of fourth page they started to care, and once they noticed it frustration just stacked (it's just a matter of adding fonts, so not a big deal). Otherwise writings are flexible enough for small variants to not be triggering (I mean, people can already read calligraphy...)
- thrdbndndn 5y ago
  Android has Japanese font. Noto Sans is basically Source Han, one of the best free JCK fonts. Not sure what you mean. https://en.wikipedia.org/wiki/Source_Han_Sans https://en.wikipedia.org/wiki/Source_Han_Sans
  - makeitdouble 5y ago
    If you buy a Xiaomi phone for instance anywhere outside Japan, and open a Japanese website, it will be displayed with Chinese glyphs. It depends on the phone but there will setup needed: sometimes just switching the whole phone language will do the deal, sometimes you need to add the right fonts yourself.
    - thrdbndndn 5y ago
      Ah that makes sense. I can imagine Chinese brands may have bad support for Japanese.
jhanschoo 5y ago
IMO the most jarring issue that comes from Han unification without proper language setting isn't with the glyph variants, since you actually encounter these variants and more in everyday life (albeit more rarely than your national standard ones). The more jarring issue is when your software selects a font meant for the wrong language, and the font for the correct language as a fallback. Then you may encounter serious style issues where your text is pockmarked by glyphs for another language; you have a similar phenomenon with European language using Latin alphabet with unconventional accents. But note that it takes way more effort to ask a CJK foundry to cover all codepoints for all languages than to ask a Latin font designer to cover all languages, so you would be hard pressed to find fonts that actually do that. Without Han unification, this wouldn't really be a problem, but Han unification is to a large extent the same philosophy pursued with unification of Latin scripts (and other scripts)
Asooka 5y ago
Out of curiosity, what do you do when you put a quote in Chinese from a Chinese author inline in Japanese text? Are you expected to write it using the Chinese forms of the characters, or do you write them using the Japanese forms? Edit: I mean what is the expected (grammatically correct) way to do it if you were writing with pen on paper.

eloisant 5y ago

What you're saying is true (although I'm not sure about 3, all the Japanese people I've talked to are annoyed by that), but it really sucks that we're dealing with problems that were supposed to be fixed by Unicode. Han unification have been a huge mistake, to save a few thousands of characters, and now we keep piling on more and more stupid emoji.

dotancohen 5y ago
> Han unification have been a huge mistake, to save a few thousands > of characters, and now we keep piling on more and more stupid emoji. This is my exact problem with Unicode. I've very grateful for the efforts that they have made in the past, but the change from "spare valuable codepoints at the expense of causing ambiguity in text" to "assign a new codepoint to every cartoon permutation of intangible nouns" is infuriating.
- oleganza 5y ago
  25 years ago computers and networks were different. Today text is a <0.1% of traffic compared with video and you have billions of bytes of RAM in every pocket computer. So yes, you can pile on more emojis and no one would be bothered. Unicode may have been never adopted at all if it had even larger set for CJK and made all CJK texts 1.5-2x larger than in Han-unified version, due to longer encoding. Also: UTF-8 did not exist yet and most systems treated text as arrays of fixed length characters.
  - anthk 5y ago
    UTF-8 existed 25 years ago, but in Plan9.
- arp242 5y ago
  Han unification and Emojis are from a different time. We're talking about a decision made about 30 years ago in the early 90s; Unicode was 16 bits (65k total codepoints) at the time. The original CJK Unified Ideographs block from 1992 consists of 21k codepoints. It was impossible to do that four times since 4×21k is more than 65k, and we're going to need space for some other languages as well. Why not make Unicode larger? Well, size was a real concern back then (still is, to some degree, but less so). Since then Unicode has expanded and now we have slightly under 1 million codepoints. Han character blocks have extended to about 93k codepoints today, and 4 times ~93k codepoints is actually feasible. But now you run in to compatibility issues: you can't remove all the old Han unification stuff (it will break text, big no-no), so you need to re-define it all anew. Is that better? How about mixing "old" Han unified codepoints with new Japanese or Chinese stuff? Will it really improve things or just cause endless confusion (see: combining characters)? For scale, all of Unicode currently defines about 145k codepoints; so even with Han unification we're talking about two thirds being taken up by just these three languages. In comparison there are currently about 3,000 emojis, although the number of codepoints is much less since many codepoints are re-used (e.g. "firefighter" is "person + firetruck", flags use the country code, etc.). In a quick check it looks like there are about 1,000 to 1,500 codepoints reserved for emojis. In comparison, this is nothing. What I'm trying to say is that the (comparatively) very low number of emojis has absolutely no bearing on this and that going off on a tangent about it is very misplaced.

gpderetta 5y ago

Is there a reason unicode doesn't have such a builtin lang tag? Similar to right-to-left and left-to-right it could help in displaying differently otherwise identical text. It could be stored in-band with the text with little changes to existing systems. The only change would be on the presentation layer, and if the tag were to be a non printable character, it would be backward compatible. An input device could implicitly tag input texts depending on the default lang. You need some form of sanitization, but you need it for right-to-left and left-to-right already.

arp242 5y ago
Actually, this exists already – there's U+E0001 (LANGUAGE TAG) and U+E007 (CANCEL TAG) and you can put a language code between those, e.g. "\uE0001ja-JP\ue007". Its use is also deprecated and discouraged. According to [1] it's often not needed, and [2] states that it puts a lot of burden on implementations and best done at a higher level such as HTTP, HTML, etc. I have no opinion on [1] as I don't speak these languages, but I do know I really hate working with these "invisible characters" in Unicode both as a user and developer. Copy an extra invisible LTR thingy or display variant codepoint and stuff can look and behave different, and it may not at all be obvious what the hell is going on (especially for those without a technical background). [1]: https://www.unicode.org/versions/Unicode14.0.0/ch05.pdf#G11552 https://www.unicode.org/versions/Unicode14.0.0/ch05.pdf#G115... [2]: https://www.unicode.org/versions/Unicode14.0.0/ch23.pdf#G30110 https://www.unicode.org/versions/Unicode14.0.0/ch23.pdf#G301...
- lifthrasiir 5y ago
  > there's U+E0001 (LANGUAGE TAG) and U+E007 (CANCEL TAG) and you can put a language code between those, e.g. "\uE0001ja-JP\ue007". "ja-JP" part is also written in tag characters, so it's actually E0001 E006A E0061 E002D E004A E0050 E007F and doesn't render even in unsupported environments.
  - arp242 5y ago
    Oh right; I never actually used it, I just know it exists.

marginalia_nu 5y ago

Re: lang="jp" My experience with web crawling is that the use of the lang-tag seems inconsistent at best. To make matters worse, sometimes content is straight up mislabeled, although Japanese sites often helpfully declare that they are using the Shift_JIS charset rather than UTF-8, which is at least somewhat helpful in figuring out that it is Japanese.

cehrlich 5y ago

I agree that for CJK-natives it's not such a big deal probably (unless they live their life in more than one of those languages). For people like me who primarily use their computer in English but also do some stuff in Japanese every now and then it's very frustrating. Of course at this point I know what's going on when 直 or whatever looks wrong, but it's still frustrating. OSs and Browsers having their own logic for it actually makes things _worse_ in some cases. Windows is especially bad (different types of UI elements care about different settings or don't care at all, so good luck having apps render correctly if you don't change your entire OS locale), and Chrome is pretty bad too, again especially on Windows. Overall MacOS/iOS and Safari does the best job by far. The failed attempt at Han Unification[1] is the worst decision the Unicode people have ever made. [1] https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification

jandrese 5y ago
I disagree. UCS-2 was the worst decision the Unicode consortium ever made. Han unification is merely a side effect of this bad decision.
nanis 5y ago
> The failed attempt at Han Unification[1] is the worst decision the Unicode people have ever made. At first I nodded my head in agreement, but then I decided I still think the failure to include separate code points for "lower case Turkish dotted I" and "upper case Turkish dotless I" is worse. You can't have 'ı' ≡ lc( uc 'ı' ) unless you already know you are processing Turkish ... completely unnecessary complication.
- naniwaduni 5y ago
  Turkish I unification, at least, wasn't a decision the Unicode people made, they inherited the mistake from earlier encodings. Given that those already existed, the alternative to having broken casefolding was, essentially, break all mixed Turkish documents transcoded from cp857 containing both "I" and "i" in non-Turkish functional directives, i.e. you'd necessarily break things like HTML documents without consistent tag casing.

needle0 5y ago

My original motive to write this page came most from issues in video games, where text is often displayed using custom routines and built-in rules in the OS & browser can't be of help. The issue crops up most often in indie games, but they can be seen even in high-profile high-budget games like Half-Life: Alyx or Resident Evil 4 VR.

makeitdouble 5y ago

> 3. Most of people are just not sensitive to such thing. I pointed it out to lots of people (when due to their setting, some glyphs are displayed wrong, like 门), and they can't care less. I think its' because most people don't deal with it in big amounts. I heard a lot more complaints from people using android phones that didn't have jp fonts by default. At the third of fourth page they started to care, and once they noticed it frustration just stacked (it's just a matter of adding fonts, so not a big deal). Otherwise writings are flexible enough for small variants to not be triggering (I mean, people can already read calligraphy...)

thrdbndndn 5y ago
Android has Japanese font. Noto Sans is basically Source Han, one of the best free JCK fonts. Not sure what you mean. https://en.wikipedia.org/wiki/Source_Han_Sans https://en.wikipedia.org/wiki/Source_Han_Sans
- makeitdouble 5y ago
  If you buy a Xiaomi phone for instance anywhere outside Japan, and open a Japanese website, it will be displayed with Chinese glyphs. It depends on the phone but there will setup needed: sometimes just switching the whole phone language will do the deal, sometimes you need to add the right fonts yourself.
  - thrdbndndn 5y ago
    Ah that makes sense. I can imagine Chinese brands may have bad support for Japanese.

jhanschoo 5y ago

IMO the most jarring issue that comes from Han unification without proper language setting isn't with the glyph variants, since you actually encounter these variants and more in everyday life (albeit more rarely than your national standard ones). The more jarring issue is when your software selects a font meant for the wrong language, and the font for the correct language as a fallback. Then you may encounter serious style issues where your text is pockmarked by glyphs for another language; you have a similar phenomenon with European language using Latin alphabet with unconventional accents. But note that it takes way more effort to ask a CJK foundry to cover all codepoints for all languages than to ask a Latin font designer to cover all languages, so you would be hard pressed to find fonts that actually do that. Without Han unification, this wouldn't really be a problem, but Han unification is to a large extent the same philosophy pursued with unification of Latin scripts (and other scripts)

Asooka 5y ago

Out of curiosity, what do you do when you put a quote in Chinese from a Chinese author inline in Japanese text? Are you expected to write it using the Chinese forms of the characters, or do you write them using the Japanese forms? Edit: I mean what is the expected (grammatically correct) way to do it if you were writing with pen on paper.