>People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation. N

>People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation. Next token prediction is the objective function. The model is asked to predict the next word yes but it's also allowed to compute the answer and more importantly, the entire training process is supposed to be the model learning and figuring out what sort of computations aid the prediction of the corpus it's trained on. If your corpus is language A followed by the translation in Language B then there's little choice but for the model to learn computations that translate as loss goes down. Is your corpus is chess moves then again, it's going to have to learn how to compute chess games to reduce loss. You can see this with toy models trained on toy problems. Example - a tiny transformer trained on addition examples - x + y = z learning an algorithm for addition. https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... "Pick the right word" is not a trivial exercise for the vast majority of text data. And again because people often make this mistake but a LLMs ultimate objective is NOT to produce "text that looks right" but "text that is right". Of course "right" as determined by the training corpus but basically anytime it picks a wrong word is opportunity for the model to learn and learn it does.