I found that the notion of token is very English centric. What you really want to do is to convert each sentence to a set of features, be it words, n-grams or p

I found that the notion of token is very English centric. What you really want to do is to convert each sentence to a set of features, be it words, n-grams or prefixes/suffixes, etc. When searching, the way I implemented was to use those features for only narrowing the candidates, and apply a post processing to filter out inexact results.