5 ms·

Katakana, Hiragana, and Unicode

111 points by zdw 4y ago

Symmetry 4y ago
I was confused by the lack of little `yx` characters there so looked it up on my own. The 'yu' in りゆ, "riyu" is 86 and the 'yu' in りゅ, "ryu", is 85.
- viggity 4y ago
  same with the small tsu which makes you kinda pause/emphasize the following consonant. Ex. あさり = asari = ah - sa - ri あっさり = assari = ah <tiny pause, hard s> sa - ri not to be confused with あつさり which is atsusari (which is a made up word), but because the tsu is regular sized, you pronounce it instead of altering other character pronunciations. Also of note - they completely left out "n" ん in hiragana, ン in katakana. And "wo" isn't really pronounced "wo", it is pronounced just "oh" and spelled "o" in romaji. And while there is a "wo" in katakana, I have never seen it used. It is used as a particle which is inherently a native japanese thing and ergo you use hiragana for it.
  - Anon1096 4y ago
    You see ヲ if you read stuff where there's a robot, alien, or super stereotypical foreigner speaking, since oftentimes their entire lines are written in katakana to feel non-native.
innocenat 4y ago
To be honest, both questions can be answered in a few seconds by looking at the code point table for Hiragana/Katakana if you already know Japanese. Hence, that's why nobody write about it. > How do the 46 characters map into the 90 characters? Because there are actually more than 46 characters. > Do they map the same way for both hiragana and katakana? Yes. That's also how we do conversion between hiragana and katakana. By adding/subtracting 0x60.
- superjan 4y ago
  For curiosity: how are they sorted? Are hiragana/katakana symbols considered equivalent?
  - innocenat 4y ago
    They are sorted by dictionary order (五十音順 gojuuonjun) and there are no duplicated symbols in hiragana-katakana blocks.
  - innocentoldguy 4y ago
    The article shows how they are sorted. Hiragana is used for things like Japanese words, particles, names, and to conjugate verbs. Katakana is used for things like foreign words, names, and sometimes emphasis. Both writing systems describe the same phonetics. For example, the hiragana か and katakana カ are both pronounced “ka.”
    - Pxtl 4y ago
      I'm surprised they're both used, from that description it sounds like one would fall by the wayside, like cursive has in North America. ... That said, culturally Japan seems like exactly the kind of place where, were they English-speakers, all the kids would absolutely be required to learn perfect cursive.
      - msbarnett 4y ago
        > I'm surprised they're both used, from that description it sounds like one would fall by the wayside, like cursive has in North America. In practice, there are no less than 4 separate scripts that are used in Japanese: hiragana, katakana, kanji, and romaji, and some mix of all 4 can appear in the same sentence. It's not so much analogous to cursive, which is a different "style" of writing the same "thing" – katakana and hiragana developed at different times for different groups and came to play different roles, and there are (usually) semantic implications to which are used.
      - layer8 4y ago
kensai 4y ago
John Cook is a genius. All his blog posts are gems. I love his job, I wish I could do it.
sylware 4y ago
Is there a plain and simple C written text shaper for unicode Katakana, Hiragana strings?
- ranger_danger 4y ago
  what is a text shaper?
  - HelloNurse 4y ago
    The software that controls how to render text to images, relying on fonts but considering higher level issues (e.g. line breaking and metrics for multiple characters) than the low-level information in a font.
- mananaysiempre 4y ago
  Modern horizontal hiragana and katakana are not complex or huge scripts, there are several dozen base characters (of one or two different widths) and two or so accent marks. There might be no spaces, you break lines whenever they run out without considering word boundaries. I expect anything capable of dealing with Latin should be able to handle this, and it hardly deserves the name of "shaping". (Adding kanji into the mix somewhat complicates matters, as there are so many potential characters you cannot just blindly cache the rasterization of every one of them and never throw any away, but that's also not the degree of complexity you get from Arabic and such.)
  - amichal 4y ago
    Line layout rules are a bit complex. Long long ago when i was 19 someone handed me a photocopied set of around 50 rules for line breaking Japanese text and followed them to implement our first draft of it in a text layout program we were adding Japanese support to. I implemented it blind, I dont speak Japanese , it never shipped and I dont remember the rules but i do remember quite some complexity around punctuation etc. This section from W3C covers some of what I remember and quite a bit more I'm sure https://www.w3.org/TR/jlreq/#line_composition_rules_for_punctuation_marks https://www.w3.org/TR/jlreq/#line_composition_rules_for_punc...
dudeinjapan 4y ago
And here's half-width katakana in non-Unicode, circa 1979: https://github.com/receipt-print-hq/escpos-printer-db/issues/70 https://github.com/receipt-print-hq/escpos-printer-db/issues... This is probably the system that is still used today by most bank printers in Japan. The code point usage was carried forward to JIS.
gweinberg 4y ago
What are these mysterious symbols that don't fit in the table?
- teshigahara 4y ago
  ゐ (wi) 𛄟 (wu) ゑ (we) 𛄠 (yi) 𛀁 (ye) ゐ and ゑ (these days pronounced the same as Japanese i and e) are known by all native Japanese speakers, were used historically, and actually still see some use in certain scenarios (like signs, or names of things). The other ones were never actually used much afaik and only recently were introduced to Unicode at all, and are probably unknown to most Japanese people except those interested in this kind of thing.
resoluteteeth 4y ago
There are a bunch of other annoying complexities with dealing with Japanese text like halfwidth/full-width characters: depending on what you're doing you may have to account for additional stuff like ｱ instead of ア, or Ａ instead of A. Ideally these wouldn't actually be used (this formatting should not be done at the character set level) but since they were included in unicode for backwards compatibility reasons, they do unfortunately get used a fair amount. Also I guess this isn't specific to Japanese, but if you use normalization in NFD form, the modifiers like handakuten get split into separate characters (I don't think most people ever use unicode normalization but iirc mac filesystem paths are normalized so it can be really confusing when you do actually run into it).
- rester324 4y ago
  Something which surprises me is that these full-width and half-width formattings are still enforced sometimes even these days in new Japanese web services. Considering that there is a seemingly easy lossless transformation path from/to these full-width/half-width alternatives, it would suffice to only support the full-width form one would think (and do some transformation as necessary without the user being involved).
  - 3np 4y ago
    My pet peeve is when the same form requires different format in different fields, ie full-width name but half-width phone number is required. That, and the insistent breaking of the back button.
  - makeitdouble 4y ago
    It's complicated. You'll want to use half-width for full number fields because html number input is half-width, and you'd need to move to the "full" alpha numeric keyboard to get full-width. Also if you allow full-width it opens the door to Chinese numerals, which you'd have to convert as well. Same thing for email address fields, allowing full-width character is user-friendly but you're pretty guaranteed to have no autocompletion on the user side, you deal with weird validations errors as some users will mistakenly convert part of the address to invalid chars, when forcing half-width pretty gets rid of that class of error altogether (assuming you don't allow full-width mail addresses in the first place. If I'm not mistaken they can be valid, including with a full-width domain) Places where there's less room for error usually allow both (like addresses for instance, where you won't run any validation on it anyway. Except on old sites and anal retentive services that will force you to go full-width sometimes)
sapkernel 4y ago
this is cool language learning project in code. Thanks
the_clarence 4y ago
I wrote a unicode library for Pascal back when I was in my 40's, it is challenging
dwg 4y ago
Many applicable references can be found on the web. Example of a more complete table: https://www.key-shortcut.com/en/writing-systems/%E3%81%B2%E3%82%89%E3%81%8C%E3%81%AA-japanese https://www.key-shortcut.com/en/writing-systems/%E3%81%B2%E3... As can be seen in this chart, the code points for Hiragana and Katakana are x60 apart (also pointed out by @innocenat). It's not clear why characters like ん, ン, and ヶ are excluded from the article. They are used extensively in everyday Japanese.
Waterluvian 4y ago
Tangential question: is it possible to evaluate the “information density” of characters of various alphabets and then evaluate, by setting aside all other important factors, the “efficiency” of an alphabet? The Greek alphabet looks, to super biased me, fairly simple in comparison to others. And yet it only has 26 characters, not 90 or whatnot. So perhaps the complexity is completely necessary. What about an alphabet that’s as minimalist as possible? Does one have to then fight with distinctiveness? Are there any examples of alphabets that are both inefficient and hard to distinguish between similar characters? Did they evolve or fall out of favour? What about data loss? How subtly “decayed” could letters be before you can’t differentiate similar ones? I’m suddenly so interested in all this…
- Symmetry 4y ago
  I'm not sure about alphabets but I'm pretty sure there are differences in written languages. All languages tend to have roughly the same bitrate when spoken in good conditions. But some languages tend to have simpler sounds spoken more rapidly, like Spanish or Japanese, and some tend to have more complicated sounds spoken more slowly, like English or Chinese. So if you compare the traditional Latin alphabet encoding of English versus Spanish then English will tend to be denser, though at the expense of a lot of pronunciation ambiguity for unfamiliar words due to having way more vowel sounds than the 5 the Latin alphabet has characters for.
- solarmist 4y ago
  Language is optimized for speech not writing. So it’s not exactly what you’re looking for, but very similar. “Human speech may have a universal transmission rate: 39 bits per second No matter how quickly you speak, you still share the same amount of information” (https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...) As for writing efficiency I would say Korean Hangul is the best writing system in the world. It combines the best attributes of alphabets and syllabaries. Also western writing systems aren’t as simple as they seem. Every letter has multiple forms upper/lower and print/cursive making at least 4x (cursive forms often vary depending of location within a word) as many forms as you expect.

111 points by zdw 4y ago

I was confused by the lack of little `yx` characters there so looked it up on my own. The 'yu' in りゆ, "riyu" is 86 and the 'yu' in りゅ, "ryu", is 85.

viggity 4y ago
same with the small tsu which makes you kinda pause/emphasize the following consonant. Ex. あさり = asari = ah - sa - ri あっさり = assari = ah <tiny pause, hard s> sa - ri not to be confused with あつさり which is atsusari (which is a made up word), but because the tsu is regular sized, you pronounce it instead of altering other character pronunciations. Also of note - they completely left out "n" ん in hiragana, ン in katakana. And "wo" isn't really pronounced "wo", it is pronounced just "oh" and spelled "o" in romaji. And while there is a "wo" in katakana, I have never seen it used. It is used as a particle which is inherently a native japanese thing and ergo you use hiragana for it.
- Anon1096 4y ago
  You see ヲ if you read stuff where there's a robot, alien, or super stereotypical foreigner speaking, since oftentimes their entire lines are written in katakana to feel non-native.

innocenat 4y ago

To be honest, both questions can be answered in a few seconds by looking at the code point table for Hiragana/Katakana if you already know Japanese. Hence, that's why nobody write about it. > How do the 46 characters map into the 90 characters? Because there are actually more than 46 characters. > Do they map the same way for both hiragana and katakana? Yes. That's also how we do conversion between hiragana and katakana. By adding/subtracting 0x60.

superjan 4y ago
For curiosity: how are they sorted? Are hiragana/katakana symbols considered equivalent?
- innocenat 4y ago
  They are sorted by dictionary order (五十音順 gojuuonjun) and there are no duplicated symbols in hiragana-katakana blocks.
- innocentoldguy 4y ago
  The article shows how they are sorted. Hiragana is used for things like Japanese words, particles, names, and to conjugate verbs. Katakana is used for things like foreign words, names, and sometimes emphasis. Both writing systems describe the same phonetics. For example, the hiragana か and katakana カ are both pronounced “ka.”
  - Pxtl 4y ago
    I'm surprised they're both used, from that description it sounds like one would fall by the wayside, like cursive has in North America. ... That said, culturally Japan seems like exactly the kind of place where, were they English-speakers, all the kids would absolutely be required to learn perfect cursive.
    - msbarnett 4y ago
      > I'm surprised they're both used, from that description it sounds like one would fall by the wayside, like cursive has in North America. In practice, there are no less than 4 separate scripts that are used in Japanese: hiragana, katakana, kanji, and romaji, and some mix of all 4 can appear in the same sentence. It's not so much analogous to cursive, which is a different "style" of writing the same "thing" – katakana and hiragana developed at different times for different groups and came to play different roles, and there are (usually) semantic implications to which are used.
    - layer8 4y ago

kensai 4y ago

John Cook is a genius. All his blog posts are gems. I love his job, I wish I could do it.

sylware 4y ago

Is there a plain and simple C written text shaper for unicode Katakana, Hiragana strings?

ranger_danger 4y ago
what is a text shaper?
- HelloNurse 4y ago
  The software that controls how to render text to images, relying on fonts but considering higher level issues (e.g. line breaking and metrics for multiple characters) than the low-level information in a font.
mananaysiempre 4y ago
Modern horizontal hiragana and katakana are not complex or huge scripts, there are several dozen base characters (of one or two different widths) and two or so accent marks. There might be no spaces, you break lines whenever they run out without considering word boundaries. I expect anything capable of dealing with Latin should be able to handle this, and it hardly deserves the name of "shaping". (Adding kanji into the mix somewhat complicates matters, as there are so many potential characters you cannot just blindly cache the rasterization of every one of them and never throw any away, but that's also not the degree of complexity you get from Arabic and such.)
- amichal 4y ago
  Line layout rules are a bit complex. Long long ago when i was 19 someone handed me a photocopied set of around 50 rules for line breaking Japanese text and followed them to implement our first draft of it in a text layout program we were adding Japanese support to. I implemented it blind, I dont speak Japanese , it never shipped and I dont remember the rules but i do remember quite some complexity around punctuation etc. This section from W3C covers some of what I remember and quite a bit more I'm sure https://www.w3.org/TR/jlreq/#line_composition_rules_for_punctuation_marks https://www.w3.org/TR/jlreq/#line_composition_rules_for_punc...

dudeinjapan 4y ago

And here's half-width katakana in non-Unicode, circa 1979: https://github.com/receipt-print-hq/escpos-printer-db/issues/70 https://github.com/receipt-print-hq/escpos-printer-db/issues... This is probably the system that is still used today by most bank printers in Japan. The code point usage was carried forward to JIS.

gweinberg 4y ago

What are these mysterious symbols that don't fit in the table?

teshigahara 4y ago
ゐ (wi) 𛄟 (wu) ゑ (we) 𛄠 (yi) 𛀁 (ye) ゐ and ゑ (these days pronounced the same as Japanese i and e) are known by all native Japanese speakers, were used historically, and actually still see some use in certain scenarios (like signs, or names of things). The other ones were never actually used much afaik and only recently were introduced to Unicode at all, and are probably unknown to most Japanese people except those interested in this kind of thing.

resoluteteeth 4y ago

There are a bunch of other annoying complexities with dealing with Japanese text like halfwidth/full-width characters: depending on what you're doing you may have to account for additional stuff like ｱ instead of ア, or Ａ instead of A. Ideally these wouldn't actually be used (this formatting should not be done at the character set level) but since they were included in unicode for backwards compatibility reasons, they do unfortunately get used a fair amount. Also I guess this isn't specific to Japanese, but if you use normalization in NFD form, the modifiers like handakuten get split into separate characters (I don't think most people ever use unicode normalization but iirc mac filesystem paths are normalized so it can be really confusing when you do actually run into it).

rester324 4y ago
Something which surprises me is that these full-width and half-width formattings are still enforced sometimes even these days in new Japanese web services. Considering that there is a seemingly easy lossless transformation path from/to these full-width/half-width alternatives, it would suffice to only support the full-width form one would think (and do some transformation as necessary without the user being involved).
- 3np 4y ago
  My pet peeve is when the same form requires different format in different fields, ie full-width name but half-width phone number is required. That, and the insistent breaking of the back button.
- makeitdouble 4y ago
  It's complicated. You'll want to use half-width for full number fields because html number input is half-width, and you'd need to move to the "full" alpha numeric keyboard to get full-width. Also if you allow full-width it opens the door to Chinese numerals, which you'd have to convert as well. Same thing for email address fields, allowing full-width character is user-friendly but you're pretty guaranteed to have no autocompletion on the user side, you deal with weird validations errors as some users will mistakenly convert part of the address to invalid chars, when forcing half-width pretty gets rid of that class of error altogether (assuming you don't allow full-width mail addresses in the first place. If I'm not mistaken they can be valid, including with a full-width domain) Places where there's less room for error usually allow both (like addresses for instance, where you won't run any validation on it anyway. Except on old sites and anal retentive services that will force you to go full-width sometimes)

sapkernel 4y ago

this is cool language learning project in code. Thanks

the_clarence 4y ago

I wrote a unicode library for Pascal back when I was in my 40's, it is challenging

dwg 4y ago

Many applicable references can be found on the web. Example of a more complete table: https://www.key-shortcut.com/en/writing-systems/%E3%81%B2%E3%82%89%E3%81%8C%E3%81%AA-japanese https://www.key-shortcut.com/en/writing-systems/%E3%81%B2%E3... As can be seen in this chart, the code points for Hiragana and Katakana are x60 apart (also pointed out by @innocenat). It's not clear why characters like ん, ン, and ヶ are excluded from the article. They are used extensively in everyday Japanese.

Waterluvian 4y ago

Tangential question: is it possible to evaluate the “information density” of characters of various alphabets and then evaluate, by setting aside all other important factors, the “efficiency” of an alphabet? The Greek alphabet looks, to super biased me, fairly simple in comparison to others. And yet it only has 26 characters, not 90 or whatnot. So perhaps the complexity is completely necessary. What about an alphabet that’s as minimalist as possible? Does one have to then fight with distinctiveness? Are there any examples of alphabets that are both inefficient and hard to distinguish between similar characters? Did they evolve or fall out of favour? What about data loss? How subtly “decayed” could letters be before you can’t differentiate similar ones? I’m suddenly so interested in all this…

Symmetry 4y ago
I'm not sure about alphabets but I'm pretty sure there are differences in written languages. All languages tend to have roughly the same bitrate when spoken in good conditions. But some languages tend to have simpler sounds spoken more rapidly, like Spanish or Japanese, and some tend to have more complicated sounds spoken more slowly, like English or Chinese. So if you compare the traditional Latin alphabet encoding of English versus Spanish then English will tend to be denser, though at the expense of a lot of pronunciation ambiguity for unfamiliar words due to having way more vowel sounds than the 5 the Latin alphabet has characters for.
solarmist 4y ago
Language is optimized for speech not writing. So it’s not exactly what you’re looking for, but very similar. “Human speech may have a universal transmission rate: 39 bits per second No matter how quickly you speak, you still share the same amount of information” (https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...) As for writing efficiency I would say Korean Hangul is the best writing system in the world. It combines the best attributes of alphabets and syllabaries. Also western writing systems aren’t as simple as they seem. Every letter has multiple forms upper/lower and print/cursive making at least 4x (cursive forms often vary depending of location within a word) as many forms as you expect.