3 ms·
Looks like it segments the dictionary and then looks it up. It could still be a problem if it oversegments, but if you look at the screenshots, most "words" are
by anewhnaccount2 8y ago
Looks like it segments the dictionary and then looks it up. It could still be a problem if it oversegments, but if you look at the screenshots, most "words" are two characters.
- KerrickStaley 8y agoThe segmentation is based on a library called jieba. It seems to work pretty well from my experience. https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba I do notice that some generated words are bogus (compounds of other words or redundant in some way). I have a file that lists redundant words, and when I notice these words I add them to the file so the tool won't generate them again. It also lists the words that they are duplicates of, so that those words can be upranked. This is the file if you're curious: https://github.com/kerrickstaley/Chinese-Vocab-List/blob/master/contrib_files/subtlex_dupes.yaml https://github.com/kerrickstaley/Chinese-Vocab-List/blob/mas...