To add to your point, note that a language model that "predicts the next token" accurately (as in predicting what human text would say) would pass the Turing t

To add to your point, note that a language model that "predicts the next token" accurately (as in predicting what human text would say) would pass the Turing test by definition. One may argue what passing the Turing test means, but at least by definition it is mimicking human intelligence in some way.