Predicting the next word is hard. To do this more efficiently than a dumb lookup table (the chinese room thought experiment) or a markov chain you basically nee

Predicting the next word is hard. To do this more efficiently than a dumb lookup table (the chinese room thought experiment) or a markov chain you basically need to build a world model. This model sits in the deeper layers of the LLMs (the deep in deep learning). Turning it into english or chinese only happens closer to the output layers. https://gwern.net/scaling-hypothesis#why-does-pretraining-work https://gwern.net/scaling-hypothesis#why-does-pretraining-wo...