Variable length markov chains would merge some states, sure, but it will still be a similar order of magnitudes. Anything longer than 4 tokens/words of context

Variable length markov chains would merge some states, sure, but it will still be a similar order of magnitudes. Anything longer than 4 tokens/words of context and you bump into 30k ^ 4-10 -> cross a billion/trillion state boundary and you lose any chance of using markov chains. Also - but here I may be wrong - there is no way to "train" markov chains to do generalisations - that is, if a given sentence didn't appear on the internet, it won't be available as a state for the chain. In this aspect they are more similar to a database than anything else.