4 ms·
Seems like they are splitting their tokens wrong. Spaces are just a suggestion. But seriously, the problems outlined reminded me very much of Japanese and kore
by wodenokoto 8y ago
Seems like they are splitting their tokens wrong. Spaces are just a suggestion.
But seriously, the problems outlined reminded me very much of Japanese and korean, just turned up to 11.
Japanese has endless amounts of homonyms and at least in theory words can go on and on and on with added conjugations. They have a lot of compound nouns, often build from abbreviations of the words compounded.
The author mentions that these compounds are a problem in Finnish NLP because the explode the size of the vocabulary.
Written Japanese does not contain any word boundaries at all. They split on morphemes for NLP tasks, which helps against exploding vocabularies, but also disseminate the parts into their own meaning-unit.
For written Japanese you have characters which can guide you in meaning for a lot of compound words, but they blow up your character space. That is not the case for Finnish, but I can't really decide if that is an advantage or disadvantage.
Also, a spellchecker and a chatbot would rely on completely different techniques, so you can have one without the other. Japanese doesn't even have spellcheckers, but they have a ton of other tools that millions of people rely on everyday for writing faster and better text, like kana-kanji conversion and word suggestions.