Once again we parade the "it just predicts the next token" around as if it was some deep insight. The magic of it all is how it does it, which we don't quite no

Once again we parade the "it just predicts the next token" around as if it was some deep insight. The magic of it all is how it does it, which we don't quite now. There might be some world model involved, there might not be, who knows. All we know is that "predicting the next token" is a simple objective that requires a lot of complexity inside the black box to work as well as it does.