LLMs are good because they are lossy. There is a simple exact model to compare against: Take the tokenized training data, put it into a Burrows–Wheeler transfo

LLMs are good because they are lossy. There is a simple exact model to compare against: Take the tokenized training data, put it into a Burrows–Wheeler transform, and add a few data structures. Then, given a context of any length, you can efficiently get the exact distribution for the next token in the training data. This is much cheaper to build and use than any LLM of comparable size. It's also much less useful, because it's overfitted to the training data. LLMs approximate this model. By losing many little details and smoothing the probability distributions, they can generalize much better beyond the training data.