I once built a Chinese search engine, which worked quite well. Stemming isn't an issue as Chinese has absolutely no inflection. The problem is that there ar

I once built a Chinese search engine, which worked quite well. Stemming isn't an issue as Chinese has absolutely no inflection. The problem is that there are no spaces between words. To separate them, I used a dictionary and a greedy algorithm. At any point in the text, I simply found the longest word in the dictionary which matched and committed to that. On rare occasions, it would get it wrong but soon recover (e.g. if my next few characters are ABCDE and the longest matching word is ABC, I'd match that, followed perhaps by D and E, even if the correct tokenization is AB CD E). Without a dictionary you can't meaningfully tokenize Chinese.