16 ms·

Sorting in Japanese – An Unsolved Problem (2011)

123 points by lelf 8y ago

aliswe 8y ago
If im not mistaken, another unsolved problem is generating url segments (or "slugs") from characters such as chinese, arabic and possibly japanese as well.
- kijin 8y ago
  You just use a (somewhat cleaned up) UTF-8 representation as a slug. UTF-8 URL components are universally supported (transparently encoded and decoded) in all modern browsers, and Google shows the real characters instead of the urlencoded version. You still end up with the urlencoded version when you hit Ctrl+C, but I expect that to be fixed in the near future as Unicode becomes even more widely used. For example, the following link should work perfectly well in all modern browsers: https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ
trw999 8y ago
https://en.m.wikipedia.org/wiki/Four-Corner_Method https://en.m.wikipedia.org/wiki/Four-Corner_Method You can sort Chinese characters (including Kanji but i'm not sure they use the Four Corners Method) by the Four Corners method. Why would you need to sort kanji phonetically in the first place? Do Japanese users actually expect names to be sorted phonetically? English speakers don't expect names to be sorted by IPA so consistency of the sorting scheme should be all that matters.
- aidenn0 8y ago
  Kanji in Japanese are sorted by writing them out phonetically in Kana and then sorting by the kana. English speakers don't expect names to be sorted by IPA because most English speakers don't know IPA, but all Japanese speakers do know the kana. The fact that they are sorted as-if written in kana means that the exact same written character will be sorted differently according to context. For "not names" we actually do have fairly good algorithms for determining the pronunciation of Kanji, but for many names there is literally no way to know how to pronounce them without asking.
- cooper12 8y ago
  I actually made a serious attempt at learning the Four-Corner Method for kanji [0] and it was very frustrating. It would be difficult to determine what parts of the kanji belonged to which corner, and which exact shape they corresponded to. And strokes wouldn't always be interpreted the way I thought they'd be since it's based on a handwritten representation of the character. Many characters also have multiple FC numbers! The FC method was never meant to uniquely identify specific characters, but just to help narrow down a list of candidates in a dictionary. Funnily enough, I also argue something similar in this thread that has the same drawbacks :) [0]: Because I was interested in typing characters while not actually knowing the kanji. The Tagaini Jisho app (https://www.tagaini.net/ https://www.tagaini.net/) was indispensable because it lets you search on multiple parameters including partial FC # and simpler methods like SKIP codes (http://nihongo.monash.edu/SKIP.html http://nihongo.monash.edu/SKIP.html). The only characters I couldn't transcribe with this method were those printed so small that the individual strokes were difficult to make out.
  - segfaultbuserr 8y ago
    Don't know about Japanese, but other well-known input schemes for Chinese includes Cangjie (https://en.wikipedia.org/wiki/Cangjie_input_method https://en.wikipedia.org/wiki/Cangjie_input_method), Zhengma (https://en.wikipedia.org/wiki/Zhengma_method https://en.wikipedia.org/wiki/Zhengma_method), and Wubi (https://en.wikipedia.org/wiki/Wubi_method https://en.wikipedia.org/wiki/Wubi_method). In fact, all of these non-phonetic input/encoding systems are highly non-intuitive and have a reputation of sharp learning curves, frustration is expected. This is because, in general, Chinese or Kanji characters are expected to be pronounced or written by the speakers, not to be indexed in a particular encoding system. Only the pronunciation is the natural form in the language. The encoding schemes are completely foreign, arbitrary to the native speakers. Using them requires extensive and systematic training. In mainland China, Hong Kong and Taiwan, in the 80-90s, learning to use a computer often starts from learning the code, and it needs at least two months of mechanical memorization to get started, and years of use to master it, just like how amateur radio operators learn Morse Code (edit: well, you don't need to memorize the code for every single character as if it's Morse Code, but remembering the standard decomposition of characters in the system is comparable to rememebr the Morse Code table). And remember, these are native speakers, much greater effort is needed for foreign speakers. Sure, schemes based on radicals have been used for a thousand years in dictionaries, but all of these schemes used today are a completely artificial creation for typing and searching things into/from computers (often on those with very limited computing power). The increase of processing power of personal computers in the late 90s allowed phonetic input systems to map pronunciation to characters heuristically, with high correctness rate. So those codes are rarely used by Chinese, and Japanese speakers (I believe) today. Unless you've learned computing in the during 80s to mid 90s, or you have a job related to language or word processing that requires typing tens of thousands of characters or creating/searching them in a language-related database, or you are someone who emphasize typing efficiency. > Because I was interested in typing characters while not actually knowing the kanji. This is actually a common requirement for people with those jobs, and one of the biggest reason to keep using them. It is especially useful when transcribing texts to computers or searching them in a database. Users also argue that, using them help preventing the modern disease of forgetting the writing of characters due to computerization, which I do see a point, similar to the spell-checker problem in English education.
- jwong_ 8y ago
  Imagine having your contact list sorted by some geometric function run on each letter of the A-Z alphabet.
  - trw999 8y ago
    That would be marginally better than memorizing ABCDEFGHIJKMNOPQRSTUVWXYZ. Is there anything inherent about the letter A that makes it get sorted in front of the letter Z?
- fenomas 8y ago
  > Do Japanese users actually expect names to be sorted phonetically? Yes - or that's how a human would sort them, at any rate.
  - trw999 8y ago
    I don't think you're in a position to speak for all of humanity.
    - fenomas 8y ago
      I meant a human as opposed to an algorithm. Point being: kanji words aren't always sorted phonetically, for the reasons described in the article, so a user may not be surprised if they aren't. But when a human is sorting kanji words they do so phonetically by reading.
  - radicsge 8y ago
    Think about numbers, it is not sorted phonetically, Even the alphabet is not. All those languages that use alphabet sort in exactly the same order regardless the pronunciation.
- Grue3 8y ago
  >Do Japanese users actually expect names to be sorted phonetically? Yes. Usually when there's a list of things you'd see subsections like あ (words that start with a vowel), か (words that start with "k" or "g"), さ (words that start with "s" or "z") and so on, and there's a specific order within each subsection.
- LeoDT 8y ago
  East-Asians usually put the last name before the first name, and most Japanese last name pronounced different, so there is no problem to them phonetically.
hermitdev 8y ago
Am I missing something subtle in this Kanji example, or all 4 names actually written the same?: "There are four Japanese women whose names you have to sort: Junko, Atsuko, Kiyoko, and Akiko. This does not seem difficult, until they each show you how they write their names in kanji: 淳子 (Junko) 淳子 (Atsuko) 淳子 (Kiyoko) 淳子 (Akiko)" I'm not familiar with Japanese at all (and have never had to deal with localization beyond date formats), even less so written, but these seem like wildly different pronunciations for some spelled the same. I know English has its own large set of warts with pronunciations and spellings (even disregarding US vs every other English speaking nation), but this seems overly odd. How do you get further context on how to pronounce a proper name like this? The post mentions context, but obviously in the above example, lacking the "abc" spelling (as the post terms it), what context do you have to know the proper pronunciation?
- segfaultbuserr 8y ago
  https://en.wikipedia.org/wiki/Japanese_name#Difficulty_of_reading_names https://en.wikipedia.org/wiki/Japanese_name#Difficulty_of_re...
- ssttevee 8y ago
  It's actually not possible. There are some cases where the kanji characters are only used for the meaning and have literally no contribution to the pronunciation. That is why every Japanese service provider will ask you for the pronunciation, without fail.
- asutekku 8y ago
  That’s why japanese people sometimes when introducing themselves also mention the kanjis because of the wildly different readings. There is literally no way knowing the reading if the user doesn’t provide it.
  - csours 8y ago
    Is this why business cards are so ubiquitous?
cooper12 8y ago
Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based on sound or shape) other than it being the convention that developed. Even hiragana can have different orderings (AIUEO vs IROHA [0]) One proposed method would be to do it how the dictionaries do it: first sort by major radical, [1] and then by stroke count. This is something most Japanese learn when learning how to write characters anyway (of course ambiguities would arise when the radical is shared and the stroke count is the same, but we could just choose a third arbitrary factor; we'd also have to decide on a specific written form as stroke count can differ depending on whether it is handwritten or the font). We could then teach our approach to schoolchildren and it would just become accepted over time like other things they learn. But wait you say, it's more natural for them to sort on pronunciation. However, if I gave you a list of polygon names and told you to sort by the number of sides they had, you'd be perfectly capable of doing it despite that not being alphabetical. Things are less "unnatural" if you grew up learning them and your brain doesn't experience dissonance. Anyway, just my hot take. [0]: https://en.wikipedia.org/wiki/Iroha https://en.wikipedia.org/wiki/Iroha [1]: https://en.wikipedia.org/wiki/Radical_(Chinese_characters) https://en.wikipedia.org/wiki/Radical_(Chinese_characters)
- mjevans 8y ago
  The concept of having a displayed value and a value to actually sort by isn't limited to Japanese words/names. It also comes up when there's a desired display function but for sorting intents different parts should be added or removed from the name of an entity. One such example is book and movie titles in a library.
- innocenat 8y ago
  > Even hiragana can have different orderings (AIUEO vs IROHA [0]) Unless otherwise mentioned, things are almost always AIUEO nowadays.
- jpatokal 8y ago
quelltext 8y ago
>In English, the logo goes with their saying: “Everything from A to Z.” This is indicated by the arrow. But in Japan, and any other country that doesn’t use English, A and Z aren’t always the first and last letters of the alphabet. This is grasping for straws. a) Many English speakers/countries are unfamiliar with the meaning of the logo's arrow. b) They will get the meaning if explained. c) Some won't because A-Z standing for "everything" requires a given level of literacy (cf. alpha and omega) d) They will get it easily when explained because it's a simple concept. Japanese people learn the English alphabet pretty early on in school and while some may not be familiar with the saying A-Z the logo and the meaning still perfectly works and would get an aha reaction when explained.
tmm84 8y ago
I've seen this problem solved a few different ways. 1) Have a romanized version of the value to sort. 2) Have a hiragana or katakana version of the value to sort (hiragana > katakana > roman order for sorting). Excel seems to sort things without telling it the reading for something. In JS and a few other languages Japanese is sorted based on UTF-8/16 code. This works for everything but kanji because of the reading requires a human.
- Technetium_Hat 8y ago
  In JavaScript, I imagine using string.localeCompare would give a more helpful sort than just UTF-8 value.
mopreme 8y ago
Blog author here. Surprised to see one of my old posts on the front page while browsing Hacker News. It's interesting to reflect on what has improved since I wrote it, and what has not. Both Android and iOS, for instance, provide mechanisms to get this right, if you know to use them and expose them for those locales (and only those locales). For example, both have a Contact object that contain corresponding phonetic-reading fields for first and last names. iOS Contact - see phoneticGivenName, phoneticFamilyName https://developer.apple.com/documentation/contacts/cncontact https://developer.apple.com/documentation/contacts/cncontact Android contact - see PHONETIC_GIVEN_NAME, PHONETIC_FAMILY_NAME https://developer.android.com/reference/android/provider/ContactsContract.CommonDataKinds.StructuredName https://developer.android.com/reference/android/provider/Con... For fun I tried using Google Translate to translate the kanji name in the post 淳子 in various contexts to see what Google thinks it is: - 淳子 translated to "Dumpling" - 淳子さん translated to "Atsuko" - 淳子様 translated to "Sadako" - 淳子さま translated to "Mrs. Lion" - 淳子殿 translated to "Mr. Reiko" - 私の名前は淳子です translated to "My name is Miko" - 私の名前は淳子です。 translated to "My name is Reiko." - 私の名前は淳子です！ translated to "My name is gyoza!" I expected them all to translate to Junko or Atsuko. The variation and unexpected results for what should be exactly the same thing is very interesting.
- deleted 8y ago
  [deleted]
- lifthrasiir 8y ago
  Many statistical machine translators like Google Translate are very sensitive to the availability of bilingual corpus. In this case GT seems to have learned that -子 means either a dumpling (餃子) or a female name ending with -ko, but haven't seen enough corpus to determine that the preceding 淳 is pronounced either Atsu- or Jun- in given context so it is guessing. Combined with the user-contributed corpus this can be rather disastrous: several machine translators had translated Japanese "初音ミク" [1] to Korean "이명박" [2] ;-) [1] https://en.wikipedia.org/wiki/Hatsune_Miku https://en.wikipedia.org/wiki/Hatsune_Miku [2] https://en.wikipedia.org/wiki/Lee_Myung-bak https://en.wikipedia.org/wiki/Lee_Myung-bak
- fenomas 8y ago
  Cool work! But my experience with Google Translate is that nowadays it seems to be wholly focused on machine-learning translations of full sentences, so for single words the results can be almost entirely random. E.g. you type in the characters for the Japanese word for "raisin", and depending on which parts you change to kanji the results can be starfish, dried fish, dried persimmons, etc. On the topic of sorting, I've always found it interesting that Japanese input forms with a pulldown menu for prefecture almost always list them in north to south order, rather than AIUEO or charcode or anything else. I think I read somewhere that this stems from some kind of official government encoding that was baked into Shift-JIS, or something along those lines.
seanlinmt 8y ago
Interesting. I didn’t know this was a problem. Kanji uses chinese characters. If sorting has been solved for chinese character, I’m assuming that the existence of Chinese dictionaries mean that sorting is no longer an issue, then why can’t the same method be used for kanji characters?
augbog 8y ago
Ran into this issue at work the other day. Relevant SO question: https://stackoverflow.com/questions/54543528/intl-collator-sorting-japanese-why-does-collator-not-prioritize-japanese-chara https://stackoverflow.com/questions/54543528/intl-collator-s...
needle0 8y ago
As a Japanese native, it pains me a lot when the westerners come upon linguistic differences like this and start calling them things like "overly odd" or "inferior" as they seem to be doing in the comments here. I can't help but smell a whiff of anglophonic condescension - I had thought political correctness have swept over the west over the last few years, but perhaps they only selectively apply to those who can talk back to you?
- deleted 8y ago
  [deleted]
- cryptnotic 8y ago
  Are you complaining about a lack of political correctness? Political correctness gives deference to people who are just a fashionable minority. If you think that the Japanese are among that group, then I’m sorry you have such a low opinion of your people.
  - tinus_hn 8y ago
    What do you mean with ‘fashionable minority’? It sounds pretty condescending.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- cmroanirgo 8y ago
  Reading the article showed me how ignorant I was to this problem... so much so that I'm surprised I haven't seen more articles like this in the past. For all the talk on ML and PC here, it's surprising that language is still a huge unsolved issue. As someoneone who knows a tiny amount of Japanese, I'm surprised that a common solution is to provide the katakana equivalent. Outside of the tech realm, is this really what a native Japanese would write on (say) a physical/ paper document: both the Kanji and Katakana?
expat2003 8y ago
Some statements you have made about the Japanese Language are ambiguous at best. Let's clear them up, but first let's be precise with the terminology. Pronunciation and sound have not interchangeable meaning. Every sound -or sequence of- in Japanese is associated with a specific character. It happens that some Kanji have multiple sounds or that a group of Kanji shares the same sounds, but still how are read it's univocal. There is no pronunciation involved. Just to give a quick example, the sound of あ is one and only one, while in English the sound of `a` has different pronunciations in pal, Paul, paediatrics, and so on. For the sake of simplicity it's ubiquitous savage practice overlapping the use of the "reading" with the one of "pronunciation" and the one of "intonation", but you should know the difference. The pronunciation transforms based on the preceding and/or following characters. There's no such concept in Japanese. The sounds of the Japanese language are distinctively dictated by Hiragana. Therefore what you have is a "Reading". A reading is made of one or more sounds. But once again readings or sounds have no multiple pronunciations. I hope I was able to shed some light on this complicated subject. It's a nuance, but makes an important impact. Quoting: "I should note that there are two different alphabetical sorting orders in Japanese. For this article I am going to use the a i u e o (あいうえお) sort order." Alphabetical order as we know it, it's one in Japan too. The order of the Hiragana works exactly like our alphabetical order: a preordered sequence of scripts from A to Z is the methodologically equivalent to the Hiragana from あ to ん, albeit there is no words starting with を and ん - last two characters of the alphabet. The Katakana alphabet is exactly like Hiragana, just the characters `design` changes - it is only used to write, in Japanese sounds, imported words from non-Japanese languages (Chinese and Korean are exception because you can use the Kanji to write them, although when writing just Chinese or Korean sounds, you will be using Katakana still). When talking about `ordering` we normally refer to which sequence we are listing the Kanji: by their Hiragana sounds is the most common way of listing them. They can also be listed by what in English are called Radicals - foundational shapes composing the Kanji ideograph, they can be listed by strokes count or be listed by their Chinese reading - the Japanese sound of the equivalent Chinese Kanji, or even by recurrence. In the Japanese language it's frequently used `ordering` by meaning, by which Kanji may or may not be listing, where only Hiragana is used to write all the words. This method is the closest rendition in Japanese to what we are accustomed to name English dictionary. More here https://en.wikipedia.org/wiki/Japanese_dictionary https://en.wikipedia.org/wiki/Japanese_dictionary I think it's already clear where the problem of sorting lays. Kanji, Hiragana and Katakana, though being different looking alphabets, have a well-defined scope and interconnection in the Japanese language. Kanji are words, thus have meaning; Hiragana offers a vocalization to the Kanji and also absolves the crucial grammatical role, providing the language with adverbs, prepositions, particles, determiner, verbs conjugation and more. Katakana usage is sidelined to just words foreign in nature, hence they exclusively represents sounds, carrying no meaning. Consequently when reading Japanese on a generic subject Kanji are mostly encountered, some Hiragana that connects them are present (it needs to be said that very common words are often written in Hiragana only, e.g. こんにちは means Hello) and sporadic Katakana when some imported word is used. This is happening because in the Japanese language there is hardly any punctuation at all - yes, there's a full stop, a comma, a way to encapsulate direct speech, but no concept of empty space. You can read Japanese from any direction you set up yourself for. Traditionally it's read top to bottom from left to right. Modern books are read as ours would be, even so sometimes you turn the pages backward to further the reading, e.g. comics etc. The English alphabet is used in case of very technical terms or for proper names - the context may sometime better served with romaji. But that's not a rule whatsoever. Katakana is employed more often. The Western alphabet -romaji in Japanese- adoption is similar to how you would take up French, Spanish, Italian or German words in your writing. (continue)
yutori 8y ago
Protip: Kirakira name
level3 8y ago
For those who want to know more about this: The article touches on just the tip of the iceberg. You might think that all you need to do is add an extra field for phonetic readings, and then simply sort on that field, but there are a lot of things that can go wrong. A naive sort (i.e. based simply on character code) will hit the following snags: 1) Hiragana vs Katakana The article focuses more on Kanji vs Kana, but Japanese users will expect Hiragana and Katakana to be properly sorted together. Either you normalize your sort field (by converting everything to Hiragana, for example) or you use a Kana-insensitive collation. 2) Half-width characters Katakana can be encoded as full-width or half-width characters (カ vs ｶ). Generally you want these treated as the same, so again you need to normalize or use a width-insensitive collation. There are also full-width alphabet characters (Ａ vs A). 3) Youon These are actual different characters (ゆ vs ゅ, つ vs っ), so you can't normalize, but you want them sorted together. Here you need a collation that's case-insensitive with respect to these. 4) Dakuten/Handakuten Like youon, these are also different characters (は vs ば vs ぱ) so you can't normalize, but you want them sorted together (insensitively). A sensitive sort will give you (はね, ばつ, ぱすた) while an insensitive sort will give you (ぱすた, ばつ, はね). There has been a lot of work around this, resulting in many different database collations over the years, some of which result in sorts that would greatly confuse Japanese users. As of today, you probably want to be using (in the case of MySQL) utf8mb4_0900_ai_ci or something similar.
- txtsd 8y ago
  I'd assume you'd want ゅ andっ to be added to the kana they're attached to, and then sorted. I'd want my き and きゅ together and び and っび together.
  - level3 8y ago
    That does make sense in a way, but I don't think that would feel natural to any native Japanese speaker. I'm not native and even I would find that ordering very odd. At the very least, it would make the sorting algorithm a lot more complex if you had to look ahead at later characters in order to sort the current prefix.
GorgeRonde 8y ago
I think taking people names as an example is extreme because when Japanese people give a name to a baby, they choose both a phonological sign and a kanji-based transcription: phonologically, they tend to pick quite a common name (like christian names in the west, contrasting it with the native-american names that are a lot more specific to the person), but try to be clever and original when it comes to writing it with kanjis. And I'm not even sure Amazon has an additional input field just for the sake of sorting names. Isn't this a common practice in the country ? (if I have to call a customer, how am I supposed to greet her if I can't pronounce her name ?)
rootsudo 8y ago
This is great, I never took this into consideration and I actually enjoy Japanese as a hobby. JLPT N4 and I'm astounded to "wow, yeah, that makes sense it's a problem." I always like unique problems like this, I never considered ABC vs Kana, Kanji -- and Romaji.
Razengan 8y ago
I'm not native to Japanese or English, but I do think Kanji is beautiful and kana is "better" than the Alphabet, and although I dislike resorting to whataboutisms† to assert a point, @everyone here who suggests the abolition of kanji, please consider this proposal for "simplifying" the English language: • Consolidate letters with identical pronunciations into a single letter: A/E, E/I, C/S, C/K, C/Q, G/J, I/Y, K/Q, U/OO, V/W, X/KS • Split letters with ambiguous/variable pronunciations into multiple letters with fixed pronunciation: A, C, E, G, I, O, S, U, X, Y • Remove any "silent" letters from all words. • Disambiguate homographs and homonyms+homographs (e.g. lie, fair.) I'm sure I missed or left out many examples, but would any of these proposals fly, at all? Do they not disregard all the subtle nuances and historical significance which only advanced learners might appreciate? Do you feel a little aghast at my naiveté for even suggesting this? What would be the response of native speakers to a foreigner campaigning for a more streamlined and consistent English? † or more accurately: Tu quoque ---- I'll just leave a piece of fun trivia as an example of the cool stuff that would be lost if kanji was abolished: The "slang" for a female ninja is kunoichi: く ku, ノ no, 一 ichi (the first two characters being katakana) and the kanji for woman is 女, made up of the following strokes: く, ノ, 一 It's something that I noticed on my own some time ago and it brought a smile to my face, like someone discovering an in-joke. There is a lot of arguably-clever wordplay like this in Japanese, and losing that would just make things bland for everyone, and for what? Just to make the language a little easier to stomach by Westerners? What we should be campaigning for, is better resources and tools for learning and looking up kanji.
ErotemeObelus 8y ago
The correct word is collation.

123 points by lelf 8y ago

If im not mistaken, another unsolved problem is generating url segments (or "slugs") from characters such as chinese, arabic and possibly japanese as well.

kijin 8y ago
You just use a (somewhat cleaned up) UTF-8 representation as a slug. UTF-8 URL components are universally supported (transparently encoded and decoded) in all modern browsers, and Google shows the real characters instead of the urlencoded version. You still end up with the urlencoded version when you hit Ctrl+C, but I expect that to be fixed in the near future as Unicode becomes even more widely used. For example, the following link should work perfectly well in all modern browsers: https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ

trw999 8y ago

https://en.m.wikipedia.org/wiki/Four-Corner_Method https://en.m.wikipedia.org/wiki/Four-Corner_Method You can sort Chinese characters (including Kanji but i'm not sure they use the Four Corners Method) by the Four Corners method. Why would you need to sort kanji phonetically in the first place? Do Japanese users actually expect names to be sorted phonetically? English speakers don't expect names to be sorted by IPA so consistency of the sorting scheme should be all that matters.

aidenn0 8y ago
Kanji in Japanese are sorted by writing them out phonetically in Kana and then sorting by the kana. English speakers don't expect names to be sorted by IPA because most English speakers don't know IPA, but all Japanese speakers do know the kana. The fact that they are sorted as-if written in kana means that the exact same written character will be sorted differently according to context. For "not names" we actually do have fairly good algorithms for determining the pronunciation of Kanji, but for many names there is literally no way to know how to pronounce them without asking.
cooper12 8y ago
I actually made a serious attempt at learning the Four-Corner Method for kanji [0] and it was very frustrating. It would be difficult to determine what parts of the kanji belonged to which corner, and which exact shape they corresponded to. And strokes wouldn't always be interpreted the way I thought they'd be since it's based on a handwritten representation of the character. Many characters also have multiple FC numbers! The FC method was never meant to uniquely identify specific characters, but just to help narrow down a list of candidates in a dictionary. Funnily enough, I also argue something similar in this thread that has the same drawbacks :) [0]: Because I was interested in typing characters while not actually knowing the kanji. The Tagaini Jisho app (https://www.tagaini.net/ https://www.tagaini.net/) was indispensable because it lets you search on multiple parameters including partial FC # and simpler methods like SKIP codes (http://nihongo.monash.edu/SKIP.html http://nihongo.monash.edu/SKIP.html). The only characters I couldn't transcribe with this method were those printed so small that the individual strokes were difficult to make out.
- segfaultbuserr 8y ago
  Don't know about Japanese, but other well-known input schemes for Chinese includes Cangjie (https://en.wikipedia.org/wiki/Cangjie_input_method https://en.wikipedia.org/wiki/Cangjie_input_method), Zhengma (https://en.wikipedia.org/wiki/Zhengma_method https://en.wikipedia.org/wiki/Zhengma_method), and Wubi (https://en.wikipedia.org/wiki/Wubi_method https://en.wikipedia.org/wiki/Wubi_method). In fact, all of these non-phonetic input/encoding systems are highly non-intuitive and have a reputation of sharp learning curves, frustration is expected. This is because, in general, Chinese or Kanji characters are expected to be pronounced or written by the speakers, not to be indexed in a particular encoding system. Only the pronunciation is the natural form in the language. The encoding schemes are completely foreign, arbitrary to the native speakers. Using them requires extensive and systematic training. In mainland China, Hong Kong and Taiwan, in the 80-90s, learning to use a computer often starts from learning the code, and it needs at least two months of mechanical memorization to get started, and years of use to master it, just like how amateur radio operators learn Morse Code (edit: well, you don't need to memorize the code for every single character as if it's Morse Code, but remembering the standard decomposition of characters in the system is comparable to rememebr the Morse Code table). And remember, these are native speakers, much greater effort is needed for foreign speakers. Sure, schemes based on radicals have been used for a thousand years in dictionaries, but all of these schemes used today are a completely artificial creation for typing and searching things into/from computers (often on those with very limited computing power). The increase of processing power of personal computers in the late 90s allowed phonetic input systems to map pronunciation to characters heuristically, with high correctness rate. So those codes are rarely used by Chinese, and Japanese speakers (I believe) today. Unless you've learned computing in the during 80s to mid 90s, or you have a job related to language or word processing that requires typing tens of thousands of characters or creating/searching them in a language-related database, or you are someone who emphasize typing efficiency. > Because I was interested in typing characters while not actually knowing the kanji. This is actually a common requirement for people with those jobs, and one of the biggest reason to keep using them. It is especially useful when transcribing texts to computers or searching them in a database. Users also argue that, using them help preventing the modern disease of forgetting the writing of characters due to computerization, which I do see a point, similar to the spell-checker problem in English education.
jwong_ 8y ago
Imagine having your contact list sorted by some geometric function run on each letter of the A-Z alphabet.
- trw999 8y ago
  That would be marginally better than memorizing ABCDEFGHIJKMNOPQRSTUVWXYZ. Is there anything inherent about the letter A that makes it get sorted in front of the letter Z?
fenomas 8y ago
> Do Japanese users actually expect names to be sorted phonetically? Yes - or that's how a human would sort them, at any rate.
- trw999 8y ago
  I don't think you're in a position to speak for all of humanity.
  - fenomas 8y ago
    I meant a human as opposed to an algorithm. Point being: kanji words aren't always sorted phonetically, for the reasons described in the article, so a user may not be surprised if they aren't. But when a human is sorting kanji words they do so phonetically by reading.
- radicsge 8y ago
  Think about numbers, it is not sorted phonetically, Even the alphabet is not. All those languages that use alphabet sort in exactly the same order regardless the pronunciation.
Grue3 8y ago
>Do Japanese users actually expect names to be sorted phonetically? Yes. Usually when there's a list of things you'd see subsections like あ (words that start with a vowel), か (words that start with "k" or "g"), さ (words that start with "s" or "z") and so on, and there's a specific order within each subsection.
LeoDT 8y ago
East-Asians usually put the last name before the first name, and most Japanese last name pronounced different, so there is no problem to them phonetically.

hermitdev 8y ago

Am I missing something subtle in this Kanji example, or all 4 names actually written the same?: "There are four Japanese women whose names you have to sort: Junko, Atsuko, Kiyoko, and Akiko. This does not seem difficult, until they each show you how they write their names in kanji: 淳子 (Junko) 淳子 (Atsuko) 淳子 (Kiyoko) 淳子 (Akiko)" I'm not familiar with Japanese at all (and have never had to deal with localization beyond date formats), even less so written, but these seem like wildly different pronunciations for some spelled the same. I know English has its own large set of warts with pronunciations and spellings (even disregarding US vs every other English speaking nation), but this seems overly odd. How do you get further context on how to pronounce a proper name like this? The post mentions context, but obviously in the above example, lacking the "abc" spelling (as the post terms it), what context do you have to know the proper pronunciation?

segfaultbuserr 8y ago
https://en.wikipedia.org/wiki/Japanese_name#Difficulty_of_reading_names https://en.wikipedia.org/wiki/Japanese_name#Difficulty_of_re...
ssttevee 8y ago
It's actually not possible. There are some cases where the kanji characters are only used for the meaning and have literally no contribution to the pronunciation. That is why every Japanese service provider will ask you for the pronunciation, without fail.
asutekku 8y ago
That’s why japanese people sometimes when introducing themselves also mention the kanjis because of the wildly different readings. There is literally no way knowing the reading if the user doesn’t provide it.
- csours 8y ago
  Is this why business cards are so ubiquitous?

cooper12 8y ago

Just a thought experiment, don't take it too seriously: The crux of the issue is that kanji don't have an inherent "natural" ordering that a user would expect. Sorting by their character code doesn't mean anything to a Japanese person. But, what if we made our own standard of what entails a "natural order". There's nothing about A–Z that makes the alphabet obligated to be in that order (and not something like based on sound or shape) other than it being the convention that developed. Even hiragana can have different orderings (AIUEO vs IROHA [0]) One proposed method would be to do it how the dictionaries do it: first sort by major radical, [1] and then by stroke count. This is something most Japanese learn when learning how to write characters anyway (of course ambiguities would arise when the radical is shared and the stroke count is the same, but we could just choose a third arbitrary factor; we'd also have to decide on a specific written form as stroke count can differ depending on whether it is handwritten or the font). We could then teach our approach to schoolchildren and it would just become accepted over time like other things they learn. But wait you say, it's more natural for them to sort on pronunciation. However, if I gave you a list of polygon names and told you to sort by the number of sides they had, you'd be perfectly capable of doing it despite that not being alphabetical. Things are less "unnatural" if you grew up learning them and your brain doesn't experience dissonance. Anyway, just my hot take. [0]: https://en.wikipedia.org/wiki/Iroha https://en.wikipedia.org/wiki/Iroha [1]: https://en.wikipedia.org/wiki/Radical_(Chinese_characters) https://en.wikipedia.org/wiki/Radical_(Chinese_characters)

mjevans 8y ago
The concept of having a displayed value and a value to actually sort by isn't limited to Japanese words/names. It also comes up when there's a desired display function but for sorting intents different parts should be added or removed from the name of an entity. One such example is book and movie titles in a library.
innocenat 8y ago
> Even hiragana can have different orderings (AIUEO vs IROHA [0]) Unless otherwise mentioned, things are almost always AIUEO nowadays.
jpatokal 8y ago

quelltext 8y ago

>In English, the logo goes with their saying: “Everything from A to Z.” This is indicated by the arrow. But in Japan, and any other country that doesn’t use English, A and Z aren’t always the first and last letters of the alphabet. This is grasping for straws. a) Many English speakers/countries are unfamiliar with the meaning of the logo's arrow. b) They will get the meaning if explained. c) Some won't because A-Z standing for "everything" requires a given level of literacy (cf. alpha and omega) d) They will get it easily when explained because it's a simple concept. Japanese people learn the English alphabet pretty early on in school and while some may not be familiar with the saying A-Z the logo and the meaning still perfectly works and would get an aha reaction when explained.

tmm84 8y ago

I've seen this problem solved a few different ways. 1) Have a romanized version of the value to sort. 2) Have a hiragana or katakana version of the value to sort (hiragana > katakana > roman order for sorting). Excel seems to sort things without telling it the reading for something. In JS and a few other languages Japanese is sorted based on UTF-8/16 code. This works for everything but kanji because of the reading requires a human.

Technetium_Hat 8y ago
In JavaScript, I imagine using string.localeCompare would give a more helpful sort than just UTF-8 value.

mopreme 8y ago

Blog author here. Surprised to see one of my old posts on the front page while browsing Hacker News. It's interesting to reflect on what has improved since I wrote it, and what has not. Both Android and iOS, for instance, provide mechanisms to get this right, if you know to use them and expose them for those locales (and only those locales). For example, both have a Contact object that contain corresponding phonetic-reading fields for first and last names. iOS Contact - see phoneticGivenName, phoneticFamilyName https://developer.apple.com/documentation/contacts/cncontact https://developer.apple.com/documentation/contacts/cncontact Android contact - see PHONETIC_GIVEN_NAME, PHONETIC_FAMILY_NAME https://developer.android.com/reference/android/provider/ContactsContract.CommonDataKinds.StructuredName https://developer.android.com/reference/android/provider/Con... For fun I tried using Google Translate to translate the kanji name in the post 淳子 in various contexts to see what Google thinks it is: - 淳子 translated to "Dumpling" - 淳子さん translated to "Atsuko" - 淳子様 translated to "Sadako" - 淳子さま translated to "Mrs. Lion" - 淳子殿 translated to "Mr. Reiko" - 私の名前は淳子です translated to "My name is Miko" - 私の名前は淳子です。 translated to "My name is Reiko." - 私の名前は淳子です！ translated to "My name is gyoza!" I expected them all to translate to Junko or Atsuko. The variation and unexpected results for what should be exactly the same thing is very interesting.

deleted 8y ago
[deleted]
lifthrasiir 8y ago
Many statistical machine translators like Google Translate are very sensitive to the availability of bilingual corpus. In this case GT seems to have learned that -子 means either a dumpling (餃子) or a female name ending with -ko, but haven't seen enough corpus to determine that the preceding 淳 is pronounced either Atsu- or Jun- in given context so it is guessing. Combined with the user-contributed corpus this can be rather disastrous: several machine translators had translated Japanese "初音ミク" [1] to Korean "이명박" [2] ;-) [1] https://en.wikipedia.org/wiki/Hatsune_Miku https://en.wikipedia.org/wiki/Hatsune_Miku [2] https://en.wikipedia.org/wiki/Lee_Myung-bak https://en.wikipedia.org/wiki/Lee_Myung-bak
fenomas 8y ago
Cool work! But my experience with Google Translate is that nowadays it seems to be wholly focused on machine-learning translations of full sentences, so for single words the results can be almost entirely random. E.g. you type in the characters for the Japanese word for "raisin", and depending on which parts you change to kanji the results can be starfish, dried fish, dried persimmons, etc. On the topic of sorting, I've always found it interesting that Japanese input forms with a pulldown menu for prefecture almost always list them in north to south order, rather than AIUEO or charcode or anything else. I think I read somewhere that this stems from some kind of official government encoding that was baked into Shift-JIS, or something along those lines.

seanlinmt 8y ago

Interesting. I didn’t know this was a problem. Kanji uses chinese characters. If sorting has been solved for chinese character, I’m assuming that the existence of Chinese dictionaries mean that sorting is no longer an issue, then why can’t the same method be used for kanji characters?

augbog 8y ago

Ran into this issue at work the other day. Relevant SO question: https://stackoverflow.com/questions/54543528/intl-collator-sorting-japanese-why-does-collator-not-prioritize-japanese-chara https://stackoverflow.com/questions/54543528/intl-collator-s...

needle0 8y ago

As a Japanese native, it pains me a lot when the westerners come upon linguistic differences like this and start calling them things like "overly odd" or "inferior" as they seem to be doing in the comments here. I can't help but smell a whiff of anglophonic condescension - I had thought political correctness have swept over the west over the last few years, but perhaps they only selectively apply to those who can talk back to you?

deleted 8y ago
[deleted]
cryptnotic 8y ago
Are you complaining about a lack of political correctness? Political correctness gives deference to people who are just a fashionable minority. If you think that the Japanese are among that group, then I’m sorry you have such a low opinion of your people.
- tinus_hn 8y ago
  What do you mean with ‘fashionable minority’? It sounds pretty condescending.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
cmroanirgo 8y ago
Reading the article showed me how ignorant I was to this problem... so much so that I'm surprised I haven't seen more articles like this in the past. For all the talk on ML and PC here, it's surprising that language is still a huge unsolved issue. As someoneone who knows a tiny amount of Japanese, I'm surprised that a common solution is to provide the katakana equivalent. Outside of the tech realm, is this really what a native Japanese would write on (say) a physical/ paper document: both the Kanji and Katakana?

expat2003 8y ago

Some statements you have made about the Japanese Language are ambiguous at best. Let's clear them up, but first let's be precise with the terminology. Pronunciation and sound have not interchangeable meaning. Every sound -or sequence of- in Japanese is associated with a specific character. It happens that some Kanji have multiple sounds or that a group of Kanji shares the same sounds, but still how are read it's univocal. There is no pronunciation involved. Just to give a quick example, the sound of あ is one and only one, while in English the sound of `a` has different pronunciations in pal, Paul, paediatrics, and so on. For the sake of simplicity it's ubiquitous savage practice overlapping the use of the "reading" with the one of "pronunciation" and the one of "intonation", but you should know the difference. The pronunciation transforms based on the preceding and/or following characters. There's no such concept in Japanese. The sounds of the Japanese language are distinctively dictated by Hiragana. Therefore what you have is a "Reading". A reading is made of one or more sounds. But once again readings or sounds have no multiple pronunciations. I hope I was able to shed some light on this complicated subject. It's a nuance, but makes an important impact. Quoting: "I should note that there are two different alphabetical sorting orders in Japanese. For this article I am going to use the a i u e o (あいうえお) sort order." Alphabetical order as we know it, it's one in Japan too. The order of the Hiragana works exactly like our alphabetical order: a preordered sequence of scripts from A to Z is the methodologically equivalent to the Hiragana from あ to ん, albeit there is no words starting with を and ん - last two characters of the alphabet. The Katakana alphabet is exactly like Hiragana, just the characters `design` changes - it is only used to write, in Japanese sounds, imported words from non-Japanese languages (Chinese and Korean are exception because you can use the Kanji to write them, although when writing just Chinese or Korean sounds, you will be using Katakana still). When talking about `ordering` we normally refer to which sequence we are listing the Kanji: by their Hiragana sounds is the most common way of listing them. They can also be listed by what in English are called Radicals - foundational shapes composing the Kanji ideograph, they can be listed by strokes count or be listed by their Chinese reading - the Japanese sound of the equivalent Chinese Kanji, or even by recurrence. In the Japanese language it's frequently used `ordering` by meaning, by which Kanji may or may not be listing, where only Hiragana is used to write all the words. This method is the closest rendition in Japanese to what we are accustomed to name English dictionary. More here https://en.wikipedia.org/wiki/Japanese_dictionary https://en.wikipedia.org/wiki/Japanese_dictionary I think it's already clear where the problem of sorting lays. Kanji, Hiragana and Katakana, though being different looking alphabets, have a well-defined scope and interconnection in the Japanese language. Kanji are words, thus have meaning; Hiragana offers a vocalization to the Kanji and also absolves the crucial grammatical role, providing the language with adverbs, prepositions, particles, determiner, verbs conjugation and more. Katakana usage is sidelined to just words foreign in nature, hence they exclusively represents sounds, carrying no meaning. Consequently when reading Japanese on a generic subject Kanji are mostly encountered, some Hiragana that connects them are present (it needs to be said that very common words are often written in Hiragana only, e.g. こんにちは means Hello) and sporadic Katakana when some imported word is used. This is happening because in the Japanese language there is hardly any punctuation at all - yes, there's a full stop, a comma, a way to encapsulate direct speech, but no concept of empty space. You can read Japanese from any direction you set up yourself for. Traditionally it's read top to bottom from left to right. Modern books are read as ours would be, even so sometimes you turn the pages backward to further the reading, e.g. comics etc. The English alphabet is used in case of very technical terms or for proper names - the context may sometime better served with romaji. But that's not a rule whatsoever. Katakana is employed more often. The Western alphabet -romaji in Japanese- adoption is similar to how you would take up French, Spanish, Italian or German words in your writing. (continue)

yutori 8y ago

Protip: Kirakira name

level3 8y ago

For those who want to know more about this: The article touches on just the tip of the iceberg. You might think that all you need to do is add an extra field for phonetic readings, and then simply sort on that field, but there are a lot of things that can go wrong. A naive sort (i.e. based simply on character code) will hit the following snags: 1) Hiragana vs Katakana The article focuses more on Kanji vs Kana, but Japanese users will expect Hiragana and Katakana to be properly sorted together. Either you normalize your sort field (by converting everything to Hiragana, for example) or you use a Kana-insensitive collation. 2) Half-width characters Katakana can be encoded as full-width or half-width characters (カ vs ｶ). Generally you want these treated as the same, so again you need to normalize or use a width-insensitive collation. There are also full-width alphabet characters (Ａ vs A). 3) Youon These are actual different characters (ゆ vs ゅ, つ vs っ), so you can't normalize, but you want them sorted together. Here you need a collation that's case-insensitive with respect to these. 4) Dakuten/Handakuten Like youon, these are also different characters (は vs ば vs ぱ) so you can't normalize, but you want them sorted together (insensitively). A sensitive sort will give you (はね, ばつ, ぱすた) while an insensitive sort will give you (ぱすた, ばつ, はね). There has been a lot of work around this, resulting in many different database collations over the years, some of which result in sorts that would greatly confuse Japanese users. As of today, you probably want to be using (in the case of MySQL) utf8mb4_0900_ai_ci or something similar.

txtsd 8y ago
I'd assume you'd want ゅ andっ to be added to the kana they're attached to, and then sorted. I'd want my き and きゅ together and び and っび together.
- level3 8y ago
  That does make sense in a way, but I don't think that would feel natural to any native Japanese speaker. I'm not native and even I would find that ordering very odd. At the very least, it would make the sorting algorithm a lot more complex if you had to look ahead at later characters in order to sort the current prefix.

GorgeRonde 8y ago

I think taking people names as an example is extreme because when Japanese people give a name to a baby, they choose both a phonological sign and a kanji-based transcription: phonologically, they tend to pick quite a common name (like christian names in the west, contrasting it with the native-american names that are a lot more specific to the person), but try to be clever and original when it comes to writing it with kanjis. And I'm not even sure Amazon has an additional input field just for the sake of sorting names. Isn't this a common practice in the country ? (if I have to call a customer, how am I supposed to greet her if I can't pronounce her name ?)

rootsudo 8y ago

This is great, I never took this into consideration and I actually enjoy Japanese as a hobby. JLPT N4 and I'm astounded to "wow, yeah, that makes sense it's a problem." I always like unique problems like this, I never considered ABC vs Kana, Kanji -- and Romaji.

Razengan 8y ago

I'm not native to Japanese or English, but I do think Kanji is beautiful and kana is "better" than the Alphabet, and although I dislike resorting to whataboutisms† to assert a point, @everyone here who suggests the abolition of kanji, please consider this proposal for "simplifying" the English language: • Consolidate letters with identical pronunciations into a single letter: A/E, E/I, C/S, C/K, C/Q, G/J, I/Y, K/Q, U/OO, V/W, X/KS • Split letters with ambiguous/variable pronunciations into multiple letters with fixed pronunciation: A, C, E, G, I, O, S, U, X, Y • Remove any "silent" letters from all words. • Disambiguate homographs and homonyms+homographs (e.g. lie, fair.) I'm sure I missed or left out many examples, but would any of these proposals fly, at all? Do they not disregard all the subtle nuances and historical significance which only advanced learners might appreciate? Do you feel a little aghast at my naiveté for even suggesting this? What would be the response of native speakers to a foreigner campaigning for a more streamlined and consistent English? † or more accurately: Tu quoque ---- I'll just leave a piece of fun trivia as an example of the cool stuff that would be lost if kanji was abolished: The "slang" for a female ninja is kunoichi: く ku, ノ no, 一 ichi (the first two characters being katakana) and the kanji for woman is 女, made up of the following strokes: く, ノ, 一 It's something that I noticed on my own some time ago and it brought a smile to my face, like someone discovering an in-joke. There is a lot of arguably-clever wordplay like this in Japanese, and losing that would just make things bland for everyone, and for what? Just to make the language a little easier to stomach by Westerners? What we should be campaigning for, is better resources and tools for learning and looking up kanji.

ErotemeObelus 8y ago

The correct word is collation.