That would be equivalent to a hidden markov chain. Those have been around for decades, but we have only managed to make them coherent for very short outputs. Ev

That would be equivalent to a hidden markov chain. Those have been around for decades, but we have only managed to make them coherent for very short outputs. Even GPT2 beats any Markov chain, so there has to be more going on Modern LLMs are able to transfer knowledge between different languages, so it's fair to assume that some mapping between human language and a more abstract internal representation happens at the input and output, instead of the model "operating" on English or Chinese or whatever language you talk with it. And once this exists, an internal "world model" (as in: a collection of facts and implications) isn't far, and seems to indeed be something most LLMs do. The reasoning on top of that world model is still very spotty though