5 ms·

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compa

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

fluffycat 8y ago
Turkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity. I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.
- szatkus 8y ago
  In Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.
isani 8y ago
Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.
- saagarjha 8y ago
  I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
  - isani 8y ago
    Sure, you can get pretty far with a fairly simple solution. But lot of the time, you get two (or more) ways to split the string into dictionary words. For a simple English example, is it "justice was served" or "just ice was served"?
    - saagarjha 8y ago
      I guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?
      - MereInterest 8y ago
        The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people. Justicehadbeenserved.
        woliveirajr 8y ago
        Please, share this with the world on tweeter.
        MereInterest 8y ago
        If you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.
  - plq 8y ago
    The stochastic strategy is to 1. enumerate every possible tag combination 2. assign a probability to each one 3. choose the parse with highest probability. 1. can be done either deterministically or stochastically. 2. requires you to have a language model trained with either human-tagged or semi-human-tagged corpus 3. was just the Viterbi algorithm last time I looked. Implementing 1 and 2 are require broad domain knowledge in two very different domains (linguistics and machine learning respectively) So while nowadays sentence segmentation can be considered a solved problem, it's far from trivial to implement one that can compete with the state of the art against real-world data. There is also a nice body of deterministic (rule-based) literature that is practically ignored nowadays.
- szatkus 8y ago
  Isn't it a case when parsing other languages from speech? Are there any audible cues between words when we speak?
- gizmo686 8y ago
  But Japanese is not written as character soup. It mixes two (actually 3) types of characters, with the "grammatical" sounds being written in hiragana and most content sounds being written in kanji. Since the grammatical sounds are a closed class, and tend to occur at word boundries, it turns out to be relativly simple to seperate words.
Grue3 8y ago
You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary. [1] https://github.com/tshatrov/ichiran https://github.com/tshatrov/ichiran
- yorwba 8y ago
  How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.
  - Grue3 8y ago
    It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior. [1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%83%E3%81%A3%E3%81%9F&r=htr https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...
    - yorwba 8y ago
      If I do echo "忘れちゃった" | mecab, I get 忘れちゃ動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる付属動詞候補（基本）反義:動詞:覚える/おぼえるった接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

fluffycat 8y ago

Turkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity. I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.

szatkus 8y ago
In Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.

isani 8y ago

Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.

saagarjha 8y ago
I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
- isani 8y ago
  Sure, you can get pretty far with a fairly simple solution. But lot of the time, you get two (or more) ways to split the string into dictionary words. For a simple English example, is it "justice was served" or "just ice was served"?
  - saagarjha 8y ago
    I guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?
    - MereInterest 8y ago
      The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people. Justicehadbeenserved.
      - woliveirajr 8y ago
        Please, share this with the world on tweeter.
        MereInterest 8y ago
        If you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.
- plq 8y ago
  The stochastic strategy is to 1. enumerate every possible tag combination 2. assign a probability to each one 3. choose the parse with highest probability. 1. can be done either deterministically or stochastically. 2. requires you to have a language model trained with either human-tagged or semi-human-tagged corpus 3. was just the Viterbi algorithm last time I looked. Implementing 1 and 2 are require broad domain knowledge in two very different domains (linguistics and machine learning respectively) So while nowadays sentence segmentation can be considered a solved problem, it's far from trivial to implement one that can compete with the state of the art against real-world data. There is also a nice body of deterministic (rule-based) literature that is practically ignored nowadays.
szatkus 8y ago
Isn't it a case when parsing other languages from speech? Are there any audible cues between words when we speak?
gizmo686 8y ago
But Japanese is not written as character soup. It mixes two (actually 3) types of characters, with the "grammatical" sounds being written in hiragana and most content sounds being written in kanji. Since the grammatical sounds are a closed class, and tend to occur at word boundries, it turns out to be relativly simple to seperate words.

Grue3 8y ago

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary. [1] https://github.com/tshatrov/ichiran https://github.com/tshatrov/ichiran

yorwba 8y ago
How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.
- Grue3 8y ago
  It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior. [1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%83%E3%81%A3%E3%81%9F&r=htr https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...
  - yorwba 8y ago
    If I do echo "忘れちゃった" | mecab, I get 忘れちゃ動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる付属動詞候補（基本）反義:動詞:覚える/おぼえるった接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.