6 ms·
Your Code Displays Japanese Wrong
- dexwiz 3y agoIt still shows a broken character for Japanese in Safari on iOS. Edit: fixed on reload.
- cratermoon 3y agoI copy/pasted the example into iTerm Build 3.4.8 and got Simplified Chinese. When copy pasted it here in this comment box (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0), I also got Simplified Chinese. What do you see? 刃直海角骨入 By the way, I do NOT have an Intel Mac, I have an Apple M1.
- deleted 3y ago[deleted]
- LoganDark 3y agoSame codepoints. Software can only choose the right font if it knows it's in the Japanese language. I'm sure if you set your browser's language to Japanese it would select the right font, but since it's probably set to English instead, it's selecting a "default" assumption for the language of those glyphs. App localization (where you specifically select "Japanese" as the language you want an app to display in) is the main target of this article, I think.
- adastra22 3y agoLinguistic identification can in principle be a bit smarter than that. Aside from the obvious "if it has kana it is Japanese" metric, there are also differences in code point choice which differentiate Simplified, Traditional, and Japanese. Fully accurate identification is certainly expensive, but quick early-out heuristics could be developed.
- rsaxvc 3y agoThis reminds me of guessing codepoint likelihood based on character distribution.
- eviks 3y agoindeed it could be better, though do you know of any apps that actually try to be smarter?
- adastra22 3y agoIn principle this is a job for Unicode font rendering libraries and/or standard OS text widgets, not app developers.
- adastra22 3y agoI'm on a Apple Silicon Mac and I see Japanese, for what it's worth.
- barryvan 3y agoI think the point of the page was that unless the site/app/program is explicit about the language being displayed, you'll get Simplified Chinese. As HN is an English-language site, it uses the default. Similarly, your terminal doesn't prioritise Japanese over Simplified or Traditional Chinese. By way of contrast, if you paste that into the search bar on https://yahoo.jp/ https://yahoo.jp/ , you'll get the appropriate Japanese characters, as that site is explicitly Japanese.
- cratermoon 3y agoInstead of explaining to me what I already know from reading the site and testing things out for myself, perhaps you could post some examples and experiences you've found.
- paradite 3y agoInterestingly I see the Japanese glyph on Apple M1. My locale is English-US and I am a native Chinese speaker. When I paste it into VS Code it also renders Japanese, but when I click on the unicode code, it goes to https://symbl.cc/en/5203/ https://symbl.cc/en/5203/ and https://symbl.cc/en/76F4/ https://symbl.cc/en/76F4/ which renders Chinese. I am typing out the Chinese equivalent for reference: 刃直 but looks like they also turned into Japanese glyph for display past Chinese input method, in fact I can't type out the Chinese glyph in any apps on macOS. It is not a big deal to me because I have no issue reading kanji as Chinese characters.
- adastra22 3y ago𝔗𝔬 𝔱𝔥𝔢 𝔍𝔞𝔭𝔞𝔫𝔢𝔰𝔢 𝔢𝔶𝔢, 𝔶𝔬𝔲𝔯 𝔱𝔢𝔵𝔱 𝔢𝔰𝔰𝔢𝔫𝔱𝔦𝔞𝔩𝔩𝔶 𝔩𝔬𝔬𝔨𝔰 𝔩𝔦𝔨𝔢 𝔱𝔥𝔦𝔰. ℜ𝔢𝔞𝔡𝔞𝔟𝔩𝔢, 𝔶𝔢𝔰, 𝔟𝔲𝔱 𝔴𝔦𝔱𝔥 𝔰𝔬𝔪𝔢 𝔰𝔦𝔤𝔫𝔦𝔣𝔦𝔠𝔞𝔫𝔱 𝔢𝔶𝔢𝔰𝔱𝔯𝔞𝔦𝔫 𝔞𝔫𝔡 𝔭𝔬𝔰𝔰𝔦𝔟𝔩𝔢 𝔠𝔬𝔫𝔣𝔲𝔰𝔦𝔬𝔫! The article doesn't focus on it as much, but it's also just as much an issue for Taiwan vs Mainland Chinese script. There's a lot of subtle font differences, and traditional Chinese rendered in a simplified Chinese font looks very much out of place, even if the right hanzi code points are used.
- lmm 3y agoThat kind of problem is much rarer, because Unicode has separate codepoints for traditional/simplified Chinese.
- klausa 3y agoIf only the article you're replying to had some example of characters that use the same code points for the same characters in Traditional and Simplified Chinese.
- lmm 3y agoThe codepoint mentioned in the article, U+5203, is a perfect example: in most of the world it will display in Simplified Chinese, whereas there's a counterpart U+2F81E that (in practice if not in theory) can be used to reliably display in Traditional Chinese. Only the Japanese rendering doesn't have a codepoint and is forced to use out-of-band methods of indicating the desired appearance.
- sirn 3y agoThis is actually less problematic on a Mac and on iOS (if you speak the language) because the rendering engine actually prioritizes CJK set according to OS' language preferences (System Settings → Language & Region → Preferred Languages). It's a much bigger issue on Linux, Android, and Windows prior to Windows 10, however.
- adastra22 3y agoUnless, like me, you have both Japanese and Chinese IMEs installed :(
- sirn 3y agoAh yes, having both probably makes the prioritization fall apart :( I have no idea if there's any sane way to make this work. Since I only speak Japanese, my fix on Linux is to prepend IPA PGothic/IPA PMincho (which only contains Japanese glyph) to lang=ja, which worked quite well.
- MiguelX413 3y agoIt works perfectly for me on Android. It even works with my phone's system language set to English, with Japanese as a fallback.
- sirn 3y agoI haven't used Android since Nokia 7 Plus (back in 2019) but the variant that I had back then (TA-1062) doesn't have Japanese locale settings, despite being Android One and all that. I had to install MoreLocale2 and modify the settings via adb to make it work. Every phone that was sold in the region I used to live (SEA) also lacks Japanese locale out-of-the-box. I had assumed this was still the case (locale availability tie to region being sold).
- MiguelX413 3y agoInteresting, I'm on the SM-G973U1. It always had a Japanese locale iirc. However, I can't recall which version of Android brought fallback locales. I don't remember if it was with this phone or my previous phone that they finally added that feature, but I'm sure it wasn't always a feature.
- FrostKiwi 3y agoYeah, quite the hassle. Even the big boys like Apple screw it up[1] [1] https://blog.frost.kiwi/joyo-kanji-unicode/ https://blog.frost.kiwi/joyo-kanji-unicode/
- Kilenaitor 3y agoWhy would Unicode bother consolidating code points like this...? Not like it's short on space.
- rsaxvc 3y agoIt was, in the old days, 16-bit, IIRC.
- Kilenaitor 3y agoAh yes. Just realized the author linked to its wikipedia page. Whoops.
- deleted 3y ago[deleted]
- kijin 3y agoThe consolidation effort would not have gone forward unless a significant number of people who have been studying these characters agreed that they were, in fact, the same characters. Many Hanji/Kanji/Hanja characters have a long history of stylistic variation and simplification for the sake of aesthetics and convenience. For linguists and historians who are used to such variations, the direction of a minor stroke that doesn't alter the meaning would seem to be a purely stylistic choice, just as serifs on Latin characters don't alter the meaning. Some variations just happen to be more popular in some regions/countries/contexts and not others. The general public, on the other hand, in each country is educated with the single "correct" variation favored by the government. Everything they read now uses the officially approved variation, so other variations look wrong. That stroke should absolutely not protrude to the other side, or you're a filthy barbarian! The Unicode consortium seems to have listened more to the linguists and historians in this case. The academic stance doesn't always fit with public perception, which is often seasoned with a large pinch of nationalism.
- lifthrasiir 3y ago> The consolidation effort would not have gone forward unless a significant number of people who have been studying these characters agreed that they were, in fact, the "same" characters. The concrete term is the "normalization rules", which dictate how given arbitrary characters are transformed into domenstic variants. As far as I know most countries with significant Han character usages already had one before Unicode, and the ROK rules were (and still are being) developed alongside with Unicode.
- charcircuit 3y agoWhy is Unicode worried about having separate character sets? Modern computers have plenty of storage space. If someone enters a form with these characters how can a site know what language to show it as. It makes no sense to work this way.
- bhawks 3y agoInitially there was a desire to use only 16 bits for all of Unicode. This would allow for only 65K characters - so space was at quite a premium. The jump from 8 bits to 16 bits effectively doubled memory costs and still wasn't enough to achieve its goal.
- charcircuit 3y agoThat was over 2 decades ago, they have had plenty of time to try and fix this.
- eviks 3y agoIt wasn't designed for modern computers, inefficiencies also show as processing overhead, not just storage overhead, and in general software bloat can overcome any hardware progress
- initplus 3y agoUnicode was originally designed with the dream of fitting everything into 16 bits. CJK unification was a controversial measure that enabled this, by combining "the same" characters together in Chinese, Japanese, and Korean. Microsoft (wchar), Sun (Java), Netscape (JS) and many others settled on a 16bit fixed width "Unicode" encoding (nowadays often labelled as UCS-2). Later it turned out that 16 bits wasn't enough anyway, but we are left with the UCS-2 mess regardless. UCS-2 and CJK unification are historical mistakes in the development of Unicode that we are stuck dealing with.
- whakim 3y agoStorage limitations from the 80's (as described by sibling comments) is one reason, but there's also another: which characters are considered their own graph (versus variants) is a somewhat fraught geopolitical issue, especially when China and Taiwan are on opposite sides of the argument. The final chapter of Yale professor Jing Tsu's Kingdom of Characters does a great job laying out how this manifests itself at meetings of Unicode working groups.
- mitthrowaway2 3y agoAh yes, this. I once downloaded an android flashcard app with a Japanese card deck for studying kanji, and was a quarter of the way through when I noticed that the characters I was memorizing had been displaying incorrectly all along.
- senkora 3y agoI had almost this exact issue in reverse when studying simplified hanzi.
- firstbabylonian 3y agoI don’t know how you can expect English speakers to care about this sort of thing, when the vast majority of them don’t even pay attention to how their own language is displayed in software they develop. Every organisation I’ve worked at meets attempts to correct apostrophes and curly quotes in their English copy with such an intense mixture of indifference, confusion, and annoyance, that anybody who tries is quickly convinced that they are the insane ones to pay attention to such inconsequential nonsense. Use of tabs vs spaces is easier to turn into a legitimate problem with business impact by comparison.
- arp242 3y ago"Curly quotes" are not somehow the "correct way" to write text. It is your preferred way, and that is something very different.
- creata 3y agoDoes this mean that Han unification was a mistake? What should software do when it accepts user content? What if a user wants to make a comment containing one quotation in Japanese and one in Chinese?
- lifthrasiir 3y agoThe current answer is "use ideographic variation selectors [1] and hope that softwares catch up soon". [1] https://en.wikipedia.org/wiki/Variant_form_(Unicode) https://en.wikipedia.org/wiki/Variant_form_(Unicode)
- e63f67dd-065b 3y agoHuh I never knew about the Han Unification project -- seeing as I only know simplified/traditional Chinese, I'm lucky enough to not have noticed any incorrect rendering in my daily usage. What's the intended solution for encoding text that uses Chinese and Japanese at the same time? For example, in an article discussing the differences between said texts? Is there some kind of hacky workaround, emoji colour selection style, or are you doomed to using pictures? This also reminds me a lot of the early "auto-detect encoding" tools long ago that tried to guess the encoding based on frequency distributions. There's a good reason to not do that to unified codepoints for CJK, but I wonder if there's a more sophisticated backwards-compatible solution here. Edit: looking at https://en.wikipedia.org/wiki/Variant_Chinese_characters https://en.wikipedia.org/wiki/Variant_Chinese_characters , some of these differences are really splitting hairs; if you gave me a text with some of the simplified characters swapped out with some of these variants, I would honestly not be able to tell the difference. There's probably more variation in the Chinese fonts out there than I can gleam from the article. I'm not convinced that some of these characters have a non-artistic difference that a fellow Chinese on the street will be able to tell apart.
- pavlov 3y ago> “What's the intended solution for encoding text that uses Chinese and Japanese at the same time?” For web documents, you can explicitly mark the language and let the browser pick the correct font. Can be applied to any element: <span lang=“ja”>…</span>
- eviks 3y agoDid Unicode consider using semantic language markers? Those could even help unify some latin alphabets and save you even more codepoints, e.g., you could have the same codepoints for Greek and English: start Greek, codepoint 'a' .... end Greek: 'a' wold be 'α' start English, codepoint 'a' ... end English: 'a' would be 'a' Could even have the same codepoint for Cyrillic alphabets
- lifthrasiir 3y ago> Did Unicode consider using semantic language markers? Once upon a time Unicode had inline language tags [1]. [1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)#Legacy_use https://en.wikipedia.org/wiki/Tags_(Unicode_block)#Legacy_us... > Those could even help unify some latin alphabets and save you even more codepoints, [...] This is a common misunderstanding; they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified.
- eviks 3y agoThanks, pity it got poorly designed and cancelled, do you know of a good document/criticism re. why that happened? The Wikipedia linked memos are process garbage docs where copyright header/footer with authors' addresses take more space than the (lack of) substance > they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified. Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints? Besides, the codepoint could be "first letter of a Language alphabet" instead of "a letter that looks like A"
- lifthrasiir 3y agoIt's not exactly "cancelled" per se, and you need to look at the correct place (which indeed takes time to learn...). At the beginning there was an Internet-Draft [1] proposing the use of invalid UTF-8 sequences, with an initial intention to be used in the Application Configuration Access Protocol (RFC 2244). Take a look at that---it is really horrible if you think about that. Nevertheless the proposal was significant because it demonstrated needs for "lightweight language tagging" and a more reasonable proposal was developed in the Unicode Technical Committee [2], which resulted in the current Tags block. Since they were meant to be used only when there is no other markup method, though, it was "born 'strongly discouraged'" from the beginning [3]. The current "deprecated" status is rather a formal confirmation of this position---[3] even clarifies that they are "indicated as 'strongly discouraged', but reserved for use with special protocols", and Unicode never removes once allocated code points, so Unicode can't do anything more about deprecated characters. Still, the intent is that you should use correct `lang="..."` attributes whenever applicable. [1] https://datatracker.ietf.org/doc/html/draft-ietf-acap-mlsf-00 https://datatracker.ietf.org/doc/html/draft-ietf-acap-mlsf-0... [2] https://www.unicode.org/L2/L1997/97255r.pdf#page=4 https://www.unicode.org/L2/L1997/97255r.pdf#page=4 [3] https://www.unicode.org/L2/L2001/01301-deprecation.txt https://www.unicode.org/L2/L2001/01301-deprecation.txt > Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints? That means you need to first locate all language tags before doing anything, which is wasteful. Almost all Unicode operations are designed to function with only local contexts in comparison.
- 0RQ2kaTBBlQp 3y agoCJK project is just a joke
- est 3y agoSo let's split CJK? Unicode is a joke anyway. It should be renamed to Multicode.
- lifthrasiir 3y agoI guess some questions may have been answered in the previous HN submission: https://news.ycombinator.com/item?id=29022906 https://news.ycombinator.com/item?id=29022906 I haven't said much about my personal feeling in that discussion, but honestly speaking as a Korean, I do feel that some Japaneses weigh too much on exact glyphs. These variant glyphs are indeed "wrong", but so does many non-standard shorthands (e.g. ⿸广マ as a shorthand of 魔) or individual preferences (e.g. the first character of Yoshinoya 𠮷野家 is an unusual variant of 吉). Why some are okay while others aren't? I'm yet to hear plausible justifications for this disparity.
- skhr0680 3y ago广+マ can stand for more than one character. That's the fatal problem with CJK unification in general. It takes the correctly entered language character X and replaces it with CJK character Y and then everyone gets confused
- lifthrasiir 3y ago⿸广マ is indeed a shorthand for multiple characters due to its phono-semantic construction, but many simplification attempts also had similar ambiguities. ⿸广マ in this sense is just a new character that has multiple meanings from the context. Z-variants merged by the CJK unification are much more subtle than this example.
- skhr0680 3y agoMy point is the same. Your text should be recorded and displayed as you entered it, especially when that can lead to confusion. > , but many simplification attempts also had similar ambiguities. No one asked to have our languages changed by Unicode
- lifthrasiir 3y agoThe first Japanese character set was JIS X 0201, which only had half-width katakanas. It is absurd in the modern standard, lacking all hiragana and kanji, and looks really awful (so much that they are used for comical effects today). I'd argue that JIS X 0201 is an extremely radical attempt compared to the Han unification---so why should the Han unification be blamed alone?
- donatj 3y agoSeveral years ago I read a post calling Han Unification essentially “Western Elitism” and made the argument that “no one” would merge the Latin alphabet and their Greek homoglphs. I disagree and am still mad about it. I ABSOLUTELY would combine them. Not doing so was an huge mistake. Every European character of the same general design should be merged. The lack of this causes so many problems and makes so many things that could be easy so much more difficult. Why do English, Spanish, German and Italian all share the same homogylhs, even though German has for instance has ß among others, Spanish has ñ, and Italian completely lacks 5e letters JKWXY. Greek or Russian though? Completely different character sets despite containing many of the same glyphs. That to me seems like “Western European Elitism” There are of course special cases where for instance the upper and lower characters don’t match across languages, but they can be resolved individually rather than as a full set.
- klausa 3y agoOut of curiosity, what languages do you speak and what is your native language?
- vore 3y agoI don't think Han unification is as evil as people make it out to be: the characters that got unified do share the same semantic meaning across languages, whereas Latin vs Cyrillic there is no semantic equivalence since letters are purely orthographic glyphs. The thing that sucks is that there's no real inline way (variation selectors, kind of) to select the language for the CJK glyph.
- sirn 3y ago> the characters that got unified do share the same semantic meaning across languages It doesn't in practice. For example: 机 in Japanese means "table", while the same codepoint in Simplified Chinese means "machine" (i.e. 機 in Japanese).
- vore 3y agoFor that example, in Chinese 机 is an alternative orthography of 几 (https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2 https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2) as in e.g. 茶几 – it's not perfect, but language is messy and I don't think Unicode did the worst job of a messy situation.
- hp6 3y agoI think the article doesn’t provide enough arguments why this is even an issue. For example what is the probability of such character being rendered incorectly in some standart tex? lets say a wikipedia article. Even more so the argumet that people don’t report this because they are "not speakers of English!” is just an assumption. Not to mention that translation applications are more than good enough for such a task.
- vore 3y agoThe issue is that the language is not accurately represented – imagine in English instead of the Latin letter "a" you see the Greek letter "α". It's still legible but it's not unreasonable to ask for an accurate depiction of a language.
- hp6 3y agoThe letter “a” has a frequency of ~8%, and it would indead be anoying, but lets say the letter “q” is rendered incorrectly which has a frequency of ~0.1% then thats just some minor issue.
- Lvl999Noob 3y agoWhile in a practical sense your argument might hold water (which I doubt, to be honest), this is not just a practical matter. This is also a matter of respect. If you think that mis-representing q is not a problem because it appears so less, then do you also think that disregarding the religious tenets of minorities is fine since they make up such a small part of the population?
- vore 3y agoI don't know why we shouldn't strive to have computers to reproduce language precisely, since computers are how we communicate most of the time and how the majority of content ends up being preserved. After all, maybe we shouldn't bother with spelling in English either, since everyone can safely approximate the meaning anyway!
- bashauma 3y ago
- thayne 3y agoAnother fun complication with this is that ttf, otf, and woff all have a limit of 2^16 glyphs. Which isn't enough to include separate glyphs for all characters in all three locales, especially if you include ligatures. This is why some fonts, such as Noto have separate font files for Chinese, Japanes, and Korean. It can also be problematic if you want to include japanese and chinese ideographs in the same document.
- mooxiu 3y agoAs a Chinese reading and writing Japanese everyday, they all LGTM
- jhanschoo 3y agoSpeaking as someone who reads Chinese and Japanese, this is not as annoying a problem as when a font designed for, say, Chinese is used to display Japanese and does not have the native-Japanese character (say, 働). Europeans using a different Latin script than a subset of English will be intimately familiar with what I am talking about. I don't think that Han unification itself is too problematic: every variant is attested to some extent in handwriting in all these societies. Furthermore, the web designer conscientious enough will 1. For applications meant to be displayed in a particular language, use a font for that language, in which case the lang-untagged content always appears right 2. For multilingual content ensure that lang tags and fonts are set up right. If the owner fails to do this, fallback font usage is going to be just as great a problem as variant issues.
- Lvl999Noob 3y agoThe main thing I noticed, wasn't the whole point of Unicode to be a single character set that could work for all languages simulateously? If the same code point is used for multiple characters in different languages, then aren't we pretty much back to code page era?
- WirelessGigabit 3y agoJust pasted in 刃直海角骨入 in Slack. And it shows the simplified Chinese version, like HN.
- butz 3y agoGeany 2.0 and gedit 45.0 are rendering Simplified Chinese instead of Japanese. How to fix this?