4 ms·

But they have "learned to implement any of the algorithms we know that we have used to make strong chess engines". Ask Claude Code to write you a chess engine.

But they have "learned to implement any of the algorithms we know that we have used to make strong chess engines". Ask Claude Code to write you a chess engine. Your objection is that they don't implement MCTS in the neurons themselves? Neither does a human, we use a C compiler when we want to play chess using MCTS. That's a separate question from whether an LLM (unaided by a C complier) can learn to play chess as well as human (also unaided by a C compiler). Certainly humans can't become grandmasters only by reading chess transcripts on the web, and certainly humans require many "thinking tokens" during a game to play effectively. Do you know for sure that a transformer can't reach grandmaster level if it is allowed to learn by playing games (as humans do) and is given a sufficient number of thinking tokens during the game? It seems near certain that they could, if someone wanted to spend the money (and I don't see why anyone would.)

YeGoblynQueenne 17d ago
Sutskever's claim is that in order to predict the next token a system must learn something about the "underying reality" that produced the token. In the context of chess that means that the LLM must learn something about playing chess (since tokens are the moves in a game of chess). My argument is that contrary to what should be expected if we take what Sutskever says to be true, they don't seem to have. Yes, I do mean that the LLM's weights are set so that it will execute minimax or MCTS when it needs to. That has nothing to do with whether humans can do the same or not. I don't disagree that a Transformer could learn to play chess if it was explicitly trained to do that. My argument is that LLMs, trained to predict the next token, have not learned to play chess. That's LLMs, not Transformers. Just to make sure this is not taken as splitting hairs, the point is that there's all sorts of claims made about what LLMs learn when they train on text. For example, there was a claim by Sundar Pichai that one of their models had learned to translate Bengali without explicitly being trained to do so. It later emerged that Bengali was indeed included in the model's training set [1]. It's not clear whether that included parallel texts, e.g. between Begnali and English or another intermediary language, in any case Sundar Pichai's claim was that the ability to translate Bengali was "emergent". So I'm interested in understanding the extent to which these "emergent" abilities are real or not. With chess, given the amount of textual data tracing games that floats about on the open internet, I would totally except some ability to play chess to "emerge". Maybe the reported 700-800 ELO level is even that sort of ability. Maybe we should only expect LLMs to learn to play at the level of an untrained, casual player. Maybe not. I have no idea. On the other hand, the fact they keep making elementary mistakes like illegal moves must be taken to mean that, so far, LLMs haven't learned to play chess. __________________ [1] https://www.buzzfeednews.com/article/pranavdixit/google-60-minutes-ai-claims-challenged https://www.buzzfeednews.com/article/pranavdixit/google-60-m...

YeGoblynQueenne 17d ago

Sutskever's claim is that in order to predict the next token a system must learn something about the "underying reality" that produced the token. In the context of chess that means that the LLM must learn something about playing chess (since tokens are the moves in a game of chess). My argument is that contrary to what should be expected if we take what Sutskever says to be true, they don't seem to have. Yes, I do mean that the LLM's weights are set so that it will execute minimax or MCTS when it needs to. That has nothing to do with whether humans can do the same or not. I don't disagree that a Transformer could learn to play chess if it was explicitly trained to do that. My argument is that LLMs, trained to predict the next token, have not learned to play chess. That's LLMs, not Transformers. Just to make sure this is not taken as splitting hairs, the point is that there's all sorts of claims made about what LLMs learn when they train on text. For example, there was a claim by Sundar Pichai that one of their models had learned to translate Bengali without explicitly being trained to do so. It later emerged that Bengali was indeed included in the model's training set [1]. It's not clear whether that included parallel texts, e.g. between Begnali and English or another intermediary language, in any case Sundar Pichai's claim was that the ability to translate Bengali was "emergent". So I'm interested in understanding the extent to which these "emergent" abilities are real or not. With chess, given the amount of textual data tracing games that floats about on the open internet, I would totally except some ability to play chess to "emerge". Maybe the reported 700-800 ELO level is even that sort of ability. Maybe we should only expect LLMs to learn to play at the level of an untrained, casual player. Maybe not. I have no idea. On the other hand, the fact they keep making elementary mistakes like illegal moves must be taken to mean that, so far, LLMs haven't learned to play chess. __________________ [1] https://www.buzzfeednews.com/article/pranavdixit/google-60-minutes-ai-claims-challenged https://www.buzzfeednews.com/article/pranavdixit/google-60-m...