3 ms·

My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, c

My strategy for tokenising Chinese is brute-force: read 5 characters ahead, is that a word? No -> 4 characters, No -> 3 characters, No -> 2 characters -> Yes, copy to output, move 2 characters ahead, read the next 5, 4, 3, 2. It's very slow. This is how humans do it though. https://pingtype.github.io/docs/advanced.html#spacing https://pingtype.github.io/docs/advanced.html#spacing If you have some Chinese text that you want to read, throw it into Pingtype (my program) and it should help you to pick up the vocabulary! https://pingtype.github.io/ https://pingtype.github.io/

yorwba 6y ago
You should be able to speed it up a bit by precomputing a prefix tree of all words. Then you can stop after three characters if there's no word in the tree that begins with those three characters. That would also allow you to do away with the 5-character limit and recognize longer words like 中华人民共和国 without a performance penalty. Humans do a bit more than simple prefix matching, though. E.g. most people won't read 有权利 as 有权/利 because that doesn't make much sense.
gruez 6y ago
>read 5 characters ahead alright programmer question: why was the magic number 5? Are there no chinese words that are longer than 5 characters? Or does 5 characters cover 99.999% of all possible cases, and is therefore "good enough"?
- peterburkimsher 6y ago
  It's the maximum limit for names on Ruten, a shopping site in Taiwan. That's also the reason I used the name 尸三十三尺 for deliveries for a while! ASCII characters weren't allowed. There are some words that are longer (e.g. Mesopotamia) but in practice those are basically all proper nouns.
yomly 6y ago
I enjoyed your first link. I remember a friend chuckling when she heard me speak chinese because I would speak "yizi yizi" which is very childlike. I still struggle neurologically to tokenize chinese in my head properly. Although I suspect it is personality driven as my music has been accused of being the same in the past!

yorwba 6y ago

You should be able to speed it up a bit by precomputing a prefix tree of all words. Then you can stop after three characters if there's no word in the tree that begins with those three characters. That would also allow you to do away with the 5-character limit and recognize longer words like 中华人民共和国 without a performance penalty. Humans do a bit more than simple prefix matching, though. E.g. most people won't read 有权利 as 有权/利 because that doesn't make much sense.

gruez 6y ago

>read 5 characters ahead alright programmer question: why was the magic number 5? Are there no chinese words that are longer than 5 characters? Or does 5 characters cover 99.999% of all possible cases, and is therefore "good enough"?

peterburkimsher 6y ago
It's the maximum limit for names on Ruten, a shopping site in Taiwan. That's also the reason I used the name 尸三十三尺 for deliveries for a while! ASCII characters weren't allowed. There are some words that are longer (e.g. Mesopotamia) but in practice those are basically all proper nouns.

yomly 6y ago

I enjoyed your first link. I remember a friend chuckling when she heard me speak chinese because I would speak "yizi yizi" which is very childlike. I still struggle neurologically to tokenize chinese in my head properly. Although I suspect it is personality driven as my music has been accused of being the same in the past!