6 ms·

How to tokenize Japanese in Python

59 points by polm23 6y ago

sova 6y ago
There have been some recent attempts at creating EBNF grammar definitions for Japanese that try to balance simplicity and coverage https://learn-japanese.org/2020/01/04/japanese-grammar-in-ebnf-notation/ https://learn-japanese.org/2020/01/04/japanese-grammar-in-eb...
- yorwba 6y ago
  If I'm not mistaken, that's a post by yourself, so you could just say "I recently attempted..."
contravariant 6y ago
It seems a bit weird that the example somehow doesn't consider 'fugashi' a compound noun and claims it's 2 nouns, 'fu kashi'. Maybe that's intended but then apparently this is pointless if you want to know how things are pronounced.
- Twisol 6y ago
  The key aspect not captured here is "rendaku" -- the voicing of a normally unvoiced consonant in a compound word. https://en.wikipedia.org/wiki/Rendaku https://en.wikipedia.org/wiki/Rendaku
  - 9nGQluzmnq3M 6y ago
    But rendaku rules are extremely complicated and often unpredictable , eg 中島 can be Nakashima or Nakajima depending in how the person chooses to pronounce it.
- polm23 6y ago
  This is part of a tradeoff that UniDic makes between "natural" word boundaries and other factors like reproducibility and usefulness for search indexes. UniDic has a rule that a "word" (or a "short unit word" 短単位, the unit used in UniDic) can't be more than two "minimal units", where an example of a minimal unit is a kanji with a Chinese reading. So 麩菓子 can't be a word, and in this case it's 菓子 with 麩 on the front. You can read more about short unit words here. https://unidic.ninjal.ac.jp/glossary#suw https://unidic.ninjal.ac.jp/glossary#suw It's not a perfect tradeoff, and I think it can be improved, but it does have its advantages.
yomly 6y ago
For someone in the know, how different is tokenization strategy between Japanese and Chinese? The grammar/sentence structure is different and Chinese does not use kana or qualifiers for things like verbs in the same way but then both have compound words and lack of spaces in text. Do languages like German which is rife with compound words have any overlap in tokenization or do you end up with completely different strategies?
- yorwba 6y ago
  The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)
  - yomly 6y ago
    Thank you - this was useful! I hadn't considered that in German compound nouns there will usually only be one way to cut the word which seems obvious now. (edge cases notwithstanding of course) Ofc in chinese (and japanese) the ambiguity is much higher. Especially when abbreviation comes into the mix
- peterburkimsher 6y ago
  My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow. This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing https://pingtype.github.io/docs/advanced.html#spacing If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/ https://pingtype.github.io/
kanobo 6y ago
If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/src/objects/js-break-iterator.cc#75 https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...
- polm23 6y ago
  So that works on similar principles to MeCab but it's more limited and performance is... not very good. I wrote about some of the issues with it recently here. https://github.com/polm/fugashi/issues/23 https://github.com/polm/fugashi/issues/23 (Also, if you learned about Chrome's Japaense tokenization on HN, that was me too!)
Danieru 6y ago
The moment I saw the title I thought of you Paul (polm). No doubt you are the world's expert on tokenizing Japanese.
ezoe 6y ago
Be careful what you do with a Japanese tokenizer. Just because you got an ability(or I'd like to call it probability since it's not perfect) to tokenize Japanese doesn't mean you can process Japanese like the language you familiar with. Please, don't do clever things. It's better to just leave Japanese string as binary blob and treat it as is, passing left to right without any modification. It is my observation that the programmer of native Indo-European languages tend to have a delusion that their native language is so fundamental that any language processing they deemed necessary must be applied to the any and all languages in the world. Which cause the locale library in most programming languages tend to have a feature that's unfit for Japanese and it break the Japanese. Yes, the presence of the locale library break the Japanese language support. These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part. People have a good intention. They want to support multiple languages. but it's a delusion that passing the string binary to locale library magically solve the problem. The locale libraries were often written badly for Japanese so it break the Japanese. I would like to call it locale-tainted. Since The locale library is part of the standard library of the programming languages and many software relies on it. We can't just avoid using locale library without massive refactoring of the source code which is impossible given the available resource for localization. So if we, the unfortunate Japanese programmer, got a job of localizing locale-tainted existing software to Japanese, we have to come up some workarounds. Modifying the source code to remove the locale library usage is the best but often not possible. Hooking locale library to replace it with Japanese-safe version are dirty workaround but it requires to reimplement most of the text manipulation which cost many man-monthes and rigorous testing, Imagine implementing a printf/regex library to replace the standard library that satisfy all the use case in the wild. The worst workaround is let the stupid locale library do whatever the stupid Japanese breaking string binary transformation they damn want, then, reverse-transform after the string binary passing through the locale library to fix the mess. A mess that was unnecessary in the first place. I was involved in C++ Standard Committee but people out there are so delusional they think it's a good idea for a new standard library to depend on broken locale library because it help the localization. Those delusional people are also designing Unicode library now. They won't fart out a good library under such delusion. I lost all hope for this industry. So please. Don't introduce stupid unnecessary language processing to the language you don't understand. Make our job less horrible.
sova 6y ago
Also cool: https://www.aclweb.org/anthology/Y03-1034.pdf https://www.aclweb.org/anthology/Y03-1034.pdf (parser from 2003 with 97% coverage on native text)

59 points by polm23 6y ago

sova 6y ago

There have been some recent attempts at creating EBNF grammar definitions for Japanese that try to balance simplicity and coverage https://learn-japanese.org/2020/01/04/japanese-grammar-in-ebnf-notation/ https://learn-japanese.org/2020/01/04/japanese-grammar-in-eb...

yorwba 6y ago
If I'm not mistaken, that's a post by yourself, so you could just say "I recently attempted..."

contravariant 6y ago

It seems a bit weird that the example somehow doesn't consider 'fugashi' a compound noun and claims it's 2 nouns, 'fu kashi'. Maybe that's intended but then apparently this is pointless if you want to know how things are pronounced.

Twisol 6y ago
The key aspect not captured here is "rendaku" -- the voicing of a normally unvoiced consonant in a compound word. https://en.wikipedia.org/wiki/Rendaku https://en.wikipedia.org/wiki/Rendaku
- 9nGQluzmnq3M 6y ago
  But rendaku rules are extremely complicated and often unpredictable , eg 中島 can be Nakashima or Nakajima depending in how the person chooses to pronounce it.
polm23 6y ago
This is part of a tradeoff that UniDic makes between "natural" word boundaries and other factors like reproducibility and usefulness for search indexes. UniDic has a rule that a "word" (or a "short unit word" 短単位, the unit used in UniDic) can't be more than two "minimal units", where an example of a minimal unit is a kanji with a Chinese reading. So 麩菓子 can't be a word, and in this case it's 菓子 with 麩 on the front. You can read more about short unit words here. https://unidic.ninjal.ac.jp/glossary#suw https://unidic.ninjal.ac.jp/glossary#suw It's not a perfect tradeoff, and I think it can be improved, but it does have its advantages.

yomly 6y ago

For someone in the know, how different is tokenization strategy between Japanese and Chinese? The grammar/sentence structure is different and Chinese does not use kana or qualifiers for things like verbs in the same way but then both have compound words and lack of spaces in text. Do languages like German which is rife with compound words have any overlap in tokenization or do you end up with completely different strategies?

yorwba 6y ago
The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)
- yomly 6y ago
  Thank you - this was useful! I hadn't considered that in German compound nouns there will usually only be one way to cut the word which seems obvious now. (edge cases notwithstanding of course) Ofc in chinese (and japanese) the ambiguity is much higher. Especially when abbreviation comes into the mix
peterburkimsher 6y ago
My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow. This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing https://pingtype.github.io/docs/advanced.html#spacing If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/ https://pingtype.github.io/

kanobo 6y ago

If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ead19613bda84621794b72/icu4j/main/classes/core/src/com/ibm/icu/text/CjkBreakEngine.java https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/src/objects/js-break-iterator.cc#75 https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...

polm23 6y ago
So that works on similar principles to MeCab but it's more limited and performance is... not very good. I wrote about some of the issues with it recently here. https://github.com/polm/fugashi/issues/23 https://github.com/polm/fugashi/issues/23 (Also, if you learned about Chrome's Japaense tokenization on HN, that was me too!)

Danieru 6y ago

The moment I saw the title I thought of you Paul (polm). No doubt you are the world's expert on tokenizing Japanese.

ezoe 6y ago

Be careful what you do with a Japanese tokenizer. Just because you got an ability(or I'd like to call it probability since it's not perfect) to tokenize Japanese doesn't mean you can process Japanese like the language you familiar with. Please, don't do clever things. It's better to just leave Japanese string as binary blob and treat it as is, passing left to right without any modification. It is my observation that the programmer of native Indo-European languages tend to have a delusion that their native language is so fundamental that any language processing they deemed necessary must be applied to the any and all languages in the world. Which cause the locale library in most programming languages tend to have a feature that's unfit for Japanese and it break the Japanese. Yes, the presence of the locale library break the Japanese language support. These locale libraries modify the string binary and break the Japanese encoding. You can't easily implement no-op local for Japanese because these modification are often cause by the fundamental common part of the library rather than language specific part. People have a good intention. They want to support multiple languages. but it's a delusion that passing the string binary to locale library magically solve the problem. The locale libraries were often written badly for Japanese so it break the Japanese. I would like to call it locale-tainted. Since The locale library is part of the standard library of the programming languages and many software relies on it. We can't just avoid using locale library without massive refactoring of the source code which is impossible given the available resource for localization. So if we, the unfortunate Japanese programmer, got a job of localizing locale-tainted existing software to Japanese, we have to come up some workarounds. Modifying the source code to remove the locale library usage is the best but often not possible. Hooking locale library to replace it with Japanese-safe version are dirty workaround but it requires to reimplement most of the text manipulation which cost many man-monthes and rigorous testing, Imagine implementing a printf/regex library to replace the standard library that satisfy all the use case in the wild. The worst workaround is let the stupid locale library do whatever the stupid Japanese breaking string binary transformation they damn want, then, reverse-transform after the string binary passing through the locale library to fix the mess. A mess that was unnecessary in the first place. I was involved in C++ Standard Committee but people out there are so delusional they think it's a good idea for a new standard library to depend on broken locale library because it help the localization. Those delusional people are also designing Unicode library now. They won't fart out a good library under such delusion. I lost all hope for this industry. So please. Don't introduce stupid unnecessary language processing to the language you don't understand. Make our job less horrible.

sova 6y ago

Also cool: https://www.aclweb.org/anthology/Y03-1034.pdf https://www.aclweb.org/anthology/Y03-1034.pdf (parser from 2003 with 97% coverage on native text)