3 ms·
The most popular tokenizer for Japanese is probably MeCab. It can also give you readings, but in cases where the reading depends on the meaning of the sentence,
by yorwba 8y ago
The most popular tokenizer for Japanese is probably MeCab. It can also give you readings, but in cases where the reading depends on the meaning of the sentence, it's about as likely to get it wrong as to get right. My current approach is to also use Kuromoji and compare the output so that I can at least notice which parts might be inaccurate.