you say it like predicting next word is an easy task. Calculating the permutations with statistics alone would probably require more compute than the entire un

you say it like predicting next word is an easy task. Calculating the permutations with statistics alone would probably require more compute than the entire universe to the power of the entire universe. And there's just not enough data, even all the internet wont be enough. We abandoned these methods long ago. Yes, LLMs really encode knowledge. This is not up for debate, and it's obvious to anyone working in AI. It outputs words and input words, yes, but the magic is what is in between. The words are broken down immediately at the first layer. Next it involves 100s of billions of parameters doing god knows what. But it's safe to assume they encode the base reality as best it can. It is what i call 'language-induced reality model'. The task of next word predictor in a humongous dataset turns out, is best solved by creating an inner representation of the real world. It's still not a perfect model since it's just induced by language though, so obviously it has limitations. But it's really modeling it, and with higher fidelity than people imagined you can learn the world through words alone.