> What does it mean to predict the next token correctly though? Arguably (non instruction tuned) models already regurgitate their training data such that it'd c

> What does it mean to predict the next token correctly though? Arguably (non instruction tuned) models already regurgitate their training data such that it'd complete "Mary had a" with "little lamb" 100% of the time. The unseen test data. Obviously omniscience is physically impossible. The point though is that the better and better next token prediction is, the more intelligent the system must be.