3 ms·
I tried making a beta version of Pingtype (Chinese/English parallel) for Japanese, but haven't figured out word spacing yet. Could you help?
by peterburkimsher 8y ago
I tried making a beta version of Pingtype (Chinese/English parallel) for Japanese, but haven't figured out word spacing yet. Could you help?
- yorwba 8y agoThe most popular tokenizer for Japanese is probably MeCab. It can also give you readings, but in cases where the reading depends on the meaning of the sentence, it's about as likely to get it wrong as to get right. My current approach is to also use Kuromoji and compare the output so that I can at least notice which parts might be inaccurate.