10 ms·

LLMs have a large quantity of chess data and still can't play for shit.

iugtmkbdfil834 8mo ago
Hm.. but do they need it.. at this point, we do have custom tools that beat humans. In a sense, all LLM need is a way to connect to that tool ( and the same is true is for counting and many other aspects ).
- Windchaser 8mo ago
  Yeah, but you know that manually telling the LLM to operate other custom tools is not going to be a long-term solution. And if an LLM could design, create, and operate a separate model, and then return/translate its results to you, that would be huge, but it also seems far away. But I'm ignorant here. Can anyone with a better background of SOTA ML tell me if this is being pursued, and if so, how far away it is? (And if not, what are the arguments against it, or what other approaches might deliver similar capacities?)
  - yunyu 8mo ago
    This has been happening for the past year on verifiable problems (did the change you made in your codebase work end-to-end, does this mathematical expression validate, did I win this chess match, etc...). The bulk of data, RL environment, and inference spend right now is on coding agents (or broadly speaking, tool use agents that can make their own tools). Recent advances in mathematical/physics research have all been with coding agents making their own "tools" by writing programs: https://openai.com/index/new-result-theoretical-physics/ https://openai.com/index/new-result-theoretical-physics/
dwohnitmok 8mo ago
Not anymore. This benchmark is for LLM chess ability: https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=readme-ov-file https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=r.... LLMs are graded according to FIDE rules so e.g. two illegal moves in a game leads to an immediate loss. This benchmark doesn't have the latest models from the last two months, but Gemini 3 (with no tools) is already at 1750 - 1800 FIDE, which is approximately probably around 1900 - 2000 USCF (about USCF expert level). This is enough to beat almost everyone at your local chess club.
- deadbabe 8mo ago
  Why do we care about this? Chess AI have long been solved problems and LLMs are just an overly brute forced approach. They will never become very efficient chess players. The correct solution is to have a conventional chess AI as a tool and use the LLM as a front end for humanized output. A software engineer who proposes just doing it all via raw LLM should be fired.
  - rodiger 8mo ago
    It's a proxy for generalized reasoning. The point isn't that LLMs are the best AI architecture for chess.
    - runarberg 8mo ago
      > It's a proxy for generalized reasoning. And so for I am only convinced that they have only succeeded on appearing to have generalized reasoning. That is, when an LLM plays chess they are performing Searle’s Chinese room thought experiment while claiming to pass the Turing test
    - deadbabe 8mo ago
      Why? Beating chess is more about searching a probability space, not reasoning. Reasoning would be more like the car wash question.
      - famouswaffles 8mo ago
        It's not entirely clear how LLMs that can play chess do so, but it is clearly very different from the way other machines do so. The construct a board, they can estimate a players skill and adjust accordingly, and unlike other machines and similarly to humans, they are sensitive to how a certain position came to be when predicting the next move. Regardless, there's plenty of reasoning in chess.
        deadbabe 8mo ago
BeetleB 8mo ago
Are you saying an LLM can't produce a chess engine that will easily beat you?
- emp17344 8mo ago
  Plagiarizing Stockfish doesn’t make me good at chess. Same principle applies.
  - cindyllm 8mo ago
    [dead]
menaerus 8mo ago
Did you already forget about the AlphaZero?
nicowesterdale 8mo ago
I wrote a, I hope, amusing breakdown of the structural reasons why off-the-shelf Large Language Models physically cannot "see" a chess board, and continue to make illegal moves, and teleport pieces as seen in Gotham Chess' latest videos. https://www.nicowesterdale.com/blog/why-llms-cant-play-chess https://www.nicowesterdale.com/blog/why-llms-cant-play-chess

Hm.. but do they need it.. at this point, we do have custom tools that beat humans. In a sense, all LLM need is a way to connect to that tool ( and the same is true is for counting and many other aspects ).

Windchaser 8mo ago
Yeah, but you know that manually telling the LLM to operate other custom tools is not going to be a long-term solution. And if an LLM could design, create, and operate a separate model, and then return/translate its results to you, that would be huge, but it also seems far away. But I'm ignorant here. Can anyone with a better background of SOTA ML tell me if this is being pursued, and if so, how far away it is? (And if not, what are the arguments against it, or what other approaches might deliver similar capacities?)
- yunyu 8mo ago
  This has been happening for the past year on verifiable problems (did the change you made in your codebase work end-to-end, does this mathematical expression validate, did I win this chess match, etc...). The bulk of data, RL environment, and inference spend right now is on coding agents (or broadly speaking, tool use agents that can make their own tools). Recent advances in mathematical/physics research have all been with coding agents making their own "tools" by writing programs: https://openai.com/index/new-result-theoretical-physics/ https://openai.com/index/new-result-theoretical-physics/

dwohnitmok 8mo ago

Not anymore. This benchmark is for LLM chess ability: https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=readme-ov-file https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=r.... LLMs are graded according to FIDE rules so e.g. two illegal moves in a game leads to an immediate loss. This benchmark doesn't have the latest models from the last two months, but Gemini 3 (with no tools) is already at 1750 - 1800 FIDE, which is approximately probably around 1900 - 2000 USCF (about USCF expert level). This is enough to beat almost everyone at your local chess club.

deadbabe 8mo ago
Why do we care about this? Chess AI have long been solved problems and LLMs are just an overly brute forced approach. They will never become very efficient chess players. The correct solution is to have a conventional chess AI as a tool and use the LLM as a front end for humanized output. A software engineer who proposes just doing it all via raw LLM should be fired.
- rodiger 8mo ago
  It's a proxy for generalized reasoning. The point isn't that LLMs are the best AI architecture for chess.
  - runarberg 8mo ago
    > It's a proxy for generalized reasoning. And so for I am only convinced that they have only succeeded on appearing to have generalized reasoning. That is, when an LLM plays chess they are performing Searle’s Chinese room thought experiment while claiming to pass the Turing test
  - deadbabe 8mo ago
    Why? Beating chess is more about searching a probability space, not reasoning. Reasoning would be more like the car wash question.
    - famouswaffles 8mo ago
      It's not entirely clear how LLMs that can play chess do so, but it is clearly very different from the way other machines do so. The construct a board, they can estimate a players skill and adjust accordingly, and unlike other machines and similarly to humans, they are sensitive to how a certain position came to be when predicting the next move. Regardless, there's plenty of reasoning in chess.
      - deadbabe 8mo ago

BeetleB 8mo ago

Are you saying an LLM can't produce a chess engine that will easily beat you?

emp17344 8mo ago
Plagiarizing Stockfish doesn’t make me good at chess. Same principle applies.
- cindyllm 8mo ago
  [dead]

menaerus 8mo ago

Did you already forget about the AlphaZero?

nicowesterdale 8mo ago

I wrote a, I hope, amusing breakdown of the structural reasons why off-the-shelf Large Language Models physically cannot "see" a chess board, and continue to make illegal moves, and teleport pieces as seen in Gotham Chess' latest videos. https://www.nicowesterdale.com/blog/why-llms-cant-play-chess https://www.nicowesterdale.com/blog/why-llms-cant-play-chess