The segmentation is based on a library called jieba. It seems to work pretty well from my experience. https://github.com/fxsjy/jieba https://github.com/fxsjy/ji

The segmentation is based on a library called jieba. It seems to work pretty well from my experience. https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba I do notice that some generated words are bogus (compounds of other words or redundant in some way). I have a file that lists redundant words, and when I notice these words I add them to the file so the tool won't generate them again. It also lists the words that they are duplicates of, so that those words can be upranked. This is the file if you're curious: https://github.com/kerrickstaley/Chinese-Vocab-List/blob/master/contrib_files/subtlex_dupes.yaml https://github.com/kerrickstaley/Chinese-Vocab-List/blob/mas...