4 ms·

For someone in the know, how different is tokenization strategy between Japanese and Chinese? The grammar/sentence structure is different and Chinese does not u

by yomly 6y ago

For someone in the know, how different is tokenization strategy between Japanese and Chinese? The grammar/sentence structure is different and Chinese does not use kana or qualifiers for things like verbs in the same way but then both have compound words and lack of spaces in text. Do languages like German which is rife with compound words have any overlap in tokenization or do you end up with completely different strategies?

yorwba 6y ago
The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)
- yomly 6y ago
  Thank you - this was useful! I hadn't considered that in German compound nouns there will usually only be one way to cut the word which seems obvious now. (edge cases notwithstanding of course) Ofc in chinese (and japanese) the ambiguity is much higher. Especially when abbreviation comes into the mix
peterburkimsher 6y ago
My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow. This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing https://pingtype.github.io/docs/advanced.html#spacing If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/ https://pingtype.github.io/
- yorwba 6y ago
  You should be able to speed it up a bit by precomputing a prefix tree of all words. Then you can stop after three characters if there's no word in the tree that begins with those three characters. That would also allow you to do away with the 5-character limit and recognize longer words like 中华人民共和国 without a performance penalty. Humans do a bit more than simple prefix matching, though. E.g. most people won't read 有权利 as 有权/利 because that doesn't make much sense.
- gruez 6y ago
  >read 5 characters ahead alright programmer question: why was the magic number 5? Are there no chinese words that are longer than 5 characters? Or does 5 characters cover 99.999% of all possible cases, and is therefore "good enough"?
  - peterburkimsher 6y ago
    It's the maximum limit for names on Ruten, a shopping site in Taiwan. That's also the reason I used the name 尸三十三尺 for deliveries for a while! ASCII characters weren't allowed. There are some words that are longer (e.g. Mesopotamia) but in practice those are basically all proper nouns.
- yomly 6y ago
  I enjoyed your first link. I remember a friend chuckling when she heard me speak chinese because I would speak "yizi yizi" which is very childlike. I still struggle neurologically to tokenize chinese in my head properly. Although I suspect it is personality driven as my music has been accused of being the same in the past!

by yomly 6y ago

yorwba 6y ago

The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)

yomly 6y ago
Thank you - this was useful! I hadn't considered that in German compound nouns there will usually only be one way to cut the word which seems obvious now. (edge cases notwithstanding of course) Ofc in chinese (and japanese) the ambiguity is much higher. Especially when abbreviation comes into the mix

peterburkimsher 6y ago

My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow. This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing https://pingtype.github.io/docs/advanced.html#spacing If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/ https://pingtype.github.io/

yorwba 6y ago
You should be able to speed it up a bit by precomputing a prefix tree of all words. Then you can stop after three characters if there's no word in the tree that begins with those three characters. That would also allow you to do away with the 5-character limit and recognize longer words like 中华人民共和国 without a performance penalty. Humans do a bit more than simple prefix matching, though. E.g. most people won't read 有权利 as 有权/利 because that doesn't make much sense.
gruez 6y ago
>read 5 characters ahead alright programmer question: why was the magic number 5? Are there no chinese words that are longer than 5 characters? Or does 5 characters cover 99.999% of all possible cases, and is therefore "good enough"?
- peterburkimsher 6y ago
  It's the maximum limit for names on Ruten, a shopping site in Taiwan. That's also the reason I used the name 尸三十三尺 for deliveries for a while! ASCII characters weren't allowed. There are some words that are longer (e.g. Mesopotamia) but in practice those are basically all proper nouns.
yomly 6y ago
I enjoyed your first link. I remember a friend chuckling when she heard me speak chinese because I would speak "yizi yizi" which is very childlike. I still struggle neurologically to tokenize chinese in my head properly. Although I suspect it is personality driven as my music has been accused of being the same in the past!