5 ms·
These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of N
by plq 8y ago
These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research.
While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology.
Also, a small terminology fix: Verbs are "conjugated" but names/objects are "declinated".
[1]: https://github.com/ahmetaa/zemberek-nlp https://github.com/ahmetaa/zemberek-nlp
- szatkus 8y agoWhat's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.
- fluffycat 8y agoTurkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity. I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.
- szatkus 8y agoIn Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.
- isani 8y agoJapanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.
- saagarjha 8y agoI have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
- isani 8y agoSure, you can get pretty far with a fairly simple solution. But lot of the time, you get two (or more) ways to split the string into dictionary words. For a simple English example, is it "justice was served" or "just ice was served"?
- saagarjha 8y agoI guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?
- MereInterest 8y agoThe murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people. Justicehadbeenserved.
- woliveirajr 8y agoPlease, share this with the world on tweeter.
- MereInterest 8y agoIf you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.
- Grue3 8y agoYou can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary. [1] https://github.com/tshatrov/ichiran https://github.com/tshatrov/ichiran
- yorwba 8y agoHow do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.
- Grue3 8y agoIt's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior. [1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%83%E3%81%A3%E3%81%9F&r=htr https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...
- yorwba 8y agoIf I do echo "忘れちゃった" | mecab, I get 忘れちゃ 動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる 付属動詞候補（基本） 反義:動詞:覚える/おぼえる った 接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.
- tallanvor 8y agoBut a spell checker is just a tiny part of the NLP domain. When people talk about NLP, they are usually referring to understanding the meaning of the entire text being analyzed, rather than just providing meanings for individual words. A spell checker can't tell you the intent or subtleties of what is being conveyed.
- plq 8y ago> But a spell checker is just a tiny part of the NLP domain. Sure, but the linked tweet claims that not even a working spellchecker exists for Finnish. I'm sure Joakim Nivre would think otherwise !