5 ms·

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently ex

by Grue3 8y ago

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary. [1] https://github.com/tshatrov/ichiran https://github.com/tshatrov/ichiran

yorwba 8y ago
How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.
- Grue3 8y ago
  It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior. [1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%83%E3%81%A3%E3%81%9F&r=htr https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...
  - yorwba 8y ago
    If I do echo "忘れちゃった" | mecab, I get 忘れちゃ動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる付属動詞候補（基本）反義:動詞:覚える/おぼえるった接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.
    - Grue3 8y ago
      That's the thing though, 忘れちゃ is a completely different, unrelated form that means 忘れて+は. So yes, this split is in fact incorrect and if it wasn't past tense, but dictionary form 忘れちゃう, the result would be even worse with う being split off as "rain"? or something.

by Grue3 8y ago

yorwba 8y ago

How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.

Grue3 8y ago
It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior. [1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%83%E3%81%A3%E3%81%9F&r=htr https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...
- yorwba 8y ago
  If I do echo "忘れちゃった" | mecab, I get 忘れちゃ動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる付属動詞候補（基本）反義:動詞:覚える/おぼえるった接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.
  - Grue3 8y ago
    That's the thing though, 忘れちゃ is a completely different, unrelated form that means 忘れて+は. So yes, this split is in fact incorrect and if it wasn't past tense, but dictionary form 忘れちゃう, the result would be even worse with う being split off as "rain"? or something.