Sutskever's claim is that in order to predict the next token a system must learn something about the "underying reality" that produced the token. In the context

Sutskever's claim is that in order to predict the next token a system must learn something about the "underying reality" that produced the token. In the context of chess that means that the LLM must learn something about playing chess (since tokens are the moves in a game of chess). My argument is that contrary to what should be expected if we take what Sutskever says to be true, they don't seem to have. Yes, I do mean that the LLM's weights are set so that it will execute minimax or MCTS when it needs to. That has nothing to do with whether humans can do the same or not. I don't disagree that a Transformer could learn to play chess if it was explicitly trained to do that. My argument is that LLMs, trained to predict the next token, have not learned to play chess. That's LLMs, not Transformers. Just to make sure this is not taken as splitting hairs, the point is that there's all sorts of claims made about what LLMs learn when they train on text. For example, there was a claim by Sundar Pichai that one of their models had learned to translate Bengali without explicitly being trained to do so. It later emerged that Bengali was indeed included in the model's training set [1]. It's not clear whether that included parallel texts, e.g. between Begnali and English or another intermediary language, in any case Sundar Pichai's claim was that the ability to translate Bengali was "emergent". So I'm interested in understanding the extent to which these "emergent" abilities are real or not. With chess, given the amount of textual data tracing games that floats about on the open internet, I would totally except some ability to play chess to "emerge". Maybe the reported 700-800 ELO level is even that sort of ability. Maybe we should only expect LLMs to learn to play at the level of an untrained, casual player. Maybe not. I have no idea. On the other hand, the fact they keep making elementary mistakes like illegal moves must be taken to mean that, so far, LLMs haven't learned to play chess. __________________ [1] https://www.buzzfeednews.com/article/pranavdixit/google-60-minutes-ai-claims-challenged https://www.buzzfeednews.com/article/pranavdixit/google-60-m...