3 ms·
You should also consider full-non-ambiguous words before trying with trigrams. "marché" is only available in French, whereas "mar", "arc", ... are available in
by redox_ 13y ago
You should also consider full-non-ambiguous words before trying with trigrams. "marché" is only available in French, whereas "mar", "arc", ... are available in lots of languages. This should drastically improve your results.
- redox_ 13y agoStore only the top N common non-ambiguous words if the RAM consumption matters ;)
- danieldk 13y agoOr store the lexicon in a determinisitic acyclic finite state automaton. E.g. (shameless plug): https://github.com/danieldk/dictomaton https://github.com/danieldk/dictomaton Though, having implemented a language guesser myself, it's only an issue with very short texts (a few words). On longer texts models based on character n-grams achieve very high accuracies.