The issue with predicting the next statement is the combinatorial infinity which is improbable to model with historical frequencies, ie., P(mat|the cat sat on

The issue with predicting the next statement is the combinatorial infinity which is improbable to model with historical frequencies, ie., P(mat|the cat sat on the) is a distribution over, say, 100k words. Whereas, P(the cat sat|on the mat) is a distribution over 100k^3 words. Part of the illusion of an LLM is that we produce create text in such a highly regular way that a mere distribution over 100k iterated, say 500 times, gives you a page of text as if modelling 100k^500.