4 ms·

Not anymore. This benchmark is for LLM chess ability: https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=readme-ov-file https://github.com/lightnesscast

Not anymore. This benchmark is for LLM chess ability: https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=readme-ov-file https://github.com/lightnesscaster/Chess-LLM-Benchmark?tab=r.... LLMs are graded according to FIDE rules so e.g. two illegal moves in a game leads to an immediate loss. This benchmark doesn't have the latest models from the last two months, but Gemini 3 (with no tools) is already at 1750 - 1800 FIDE, which is approximately probably around 1900 - 2000 USCF (about USCF expert level). This is enough to beat almost everyone at your local chess club.

deadbabe 8mo ago
Why do we care about this? Chess AI have long been solved problems and LLMs are just an overly brute forced approach. They will never become very efficient chess players. The correct solution is to have a conventional chess AI as a tool and use the LLM as a front end for humanized output. A software engineer who proposes just doing it all via raw LLM should be fired.
- rodiger 8mo ago
  It's a proxy for generalized reasoning. The point isn't that LLMs are the best AI architecture for chess.
  - runarberg 8mo ago
    > It's a proxy for generalized reasoning. And so for I am only convinced that they have only succeeded on appearing to have generalized reasoning. That is, when an LLM plays chess they are performing Searle’s Chinese room thought experiment while claiming to pass the Turing test
  - deadbabe 8mo ago
    Why? Beating chess is more about searching a probability space, not reasoning. Reasoning would be more like the car wash question.
    - famouswaffles 8mo ago
      It's not entirely clear how LLMs that can play chess do so, but it is clearly very different from the way other machines do so. The construct a board, they can estimate a players skill and adjust accordingly, and unlike other machines and similarly to humans, they are sensitive to how a certain position came to be when predicting the next move. Regardless, there's plenty of reasoning in chess.
      - deadbabe 8mo ago
        It’s very clear how, chess moves and positions are vector encoded into their training data, when they are prompted with a certain board state, they respond with the most probable response to that. There is no reason.
        famouswaffles 8mo ago
        Actual Researchers can't give you a complete answer but you can. Whatever you say.
        deadbabe 8mo ago
        Guess what, I’m a researcher. I have published papers.
        famouswaffles 8mo ago
        Okay
cesarvarela 8mo ago
Yeah, but 1800 FIDE players don't make illegal moves, and Gemini does.
- famouswaffles 8mo ago
  That benchmark methodology isn't great, but regardless, LLMs can be trained to play Chess with a 99.8% legal move rate.
  - recursive 8mo ago
    That doesn't exactly sound like strong chess play.
    - dwohnitmok 8mo ago
      It's enough to reliably beat amateur (e.g. maia-1900) chess engines.
- dwohnitmok 8mo ago
  1800 FIDE players do make illegal moves. I believe they make about one to two orders of magnitude less illegal moves than Gemini 3 does here. IIRC the usual statistic for expert chess play is about 0.02% of expert chess games have an illegal move (I can look that up later if there's interest to be sure), but that is only the ones that made it into the final game notation (and weren't e.g. corrected at the board by an opponent or arbiter). So that should be a lower bound (hence why it could be up to one order lower, although I suspect two orders is still probably closer to the truth). Whether or not we'll see LLMs continue to get a lower error rate to make up for those orders of magnitude remains to be seen (I could see it go either way in the next two years based on the current rate of progress).
  - cesarvarela 8mo ago
    A player at that level making an illegal move is either tired, distracted, drunk, etc. An LLM makes it because it does not really "understand" the rules of chess.
runarberg 8mo ago
Wait, I may be missing something here. These benchmarks are gathered by having models play each other, and the second illegal move forfeits the game. This seems like a flawed method as the models who are more prone to illegal moves are going to bump the ratings of the models who are less likely. Additionally, how do we know the model isn’t benchmaxxed to eliminate illegal moves. For example, here is the list of games by Gemini-3-pro-preview. In 44 games it preformed 3 illegal moves (if I counted correctly) but won 5 because opponent forfeits due to illegal moves. https://chessbenchllm.onrender.com/games?page=5&model=gemini-3-pro-preview https://chessbenchllm.onrender.com/games?page=5&model=gemini... I suspect the ratings here may be significantly inflated due to a flaw in the methodology. EDIT: I want to suggest a better methodology here (I am not gonna do it; I really really really don’t care about this technology). Have the LLMs play rated engines and rated humans, the first illegal move forfeits the game (same rules apply to humans).
- emp17344 8mo ago
  That’s a devastating benchmark design flaw. Sick of these bullshit benchmarks designed solely to hype AI. AI boosters turn around and use them as ammo, despite not understanding them.
  - runarberg 8mo ago
    I like this game between grok-4.1-fast and maia-1100 (engine, not LLM). https://chessbenchllm.onrender.com/game/37d0d260-d63b-4e41-9bba-6b25b922493f https://chessbenchllm.onrender.com/game/37d0d260-d63b-4e41-9... This exact game has been played 60 thousand times on lichess. The peace sacrifice Grok performed on move 6 has been played 5 million times on lichess. Every single move Grok made is also the top played move on lichess. This reminds me of Stefan Zweig’s The Royal Game where the protagonist survived Nazi torture by memorizing every game in a chess book his torturers dropped (excellent book btw. and I am aware I just committed Godwin’s law here; also aware of the irony here). The protagonist became “good” at chess, simply by memorizing a lot of games.
    - famouswaffles 8mo ago
overgard 8mo ago
They have literally every chess game in existence to train on, and they can't do better than 1800?
- jimbokun 8mo ago
  Why do you think they won’t continue to improve?
  - overgard 8mo ago
    Because of how LLM's work. I don't know exactly how they're using it for chess, but here's a guess. If you consider the chess game a "conversation" between two opponents, the moves written out would be the context window. So you're asking the LLM, "given these last 30 moves, what's the most likely next move?". Ie, you're giving it a string like "1. e4 e5, 2. Nf3 Nc6, 3. Bb5 a6, 4..?". That's basically what you're doing with LLMs in any context "Here's a set of tokens, what's the most likely continuation?". The problem is, that's the wrong question for a chess move. If you're going with "most likely continuation", that will work great for openings and well-studied move sequences (there are a lot of well studied move sequences!), however, once the game becomes "a brand new game", as chess streamers like to say when there's no longer a game in the database with that set of moves, then "what's the most likely continuation from this position?" is not the right question. Non-LLM AI's have obviously solved chess, so, it doesn't really matter -- I think Chess shows how LLM's lack of a world model as Gary Marcus would say is a problem.

deadbabe 8mo ago

Why do we care about this? Chess AI have long been solved problems and LLMs are just an overly brute forced approach. They will never become very efficient chess players. The correct solution is to have a conventional chess AI as a tool and use the LLM as a front end for humanized output. A software engineer who proposes just doing it all via raw LLM should be fired.

rodiger 8mo ago
It's a proxy for generalized reasoning. The point isn't that LLMs are the best AI architecture for chess.
- runarberg 8mo ago
  > It's a proxy for generalized reasoning. And so for I am only convinced that they have only succeeded on appearing to have generalized reasoning. That is, when an LLM plays chess they are performing Searle’s Chinese room thought experiment while claiming to pass the Turing test
- deadbabe 8mo ago
  Why? Beating chess is more about searching a probability space, not reasoning. Reasoning would be more like the car wash question.
  - famouswaffles 8mo ago
    It's not entirely clear how LLMs that can play chess do so, but it is clearly very different from the way other machines do so. The construct a board, they can estimate a players skill and adjust accordingly, and unlike other machines and similarly to humans, they are sensitive to how a certain position came to be when predicting the next move. Regardless, there's plenty of reasoning in chess.
    - deadbabe 8mo ago
      It’s very clear how, chess moves and positions are vector encoded into their training data, when they are prompted with a certain board state, they respond with the most probable response to that. There is no reason.
      - famouswaffles 8mo ago
        Actual Researchers can't give you a complete answer but you can. Whatever you say.
        deadbabe 8mo ago
        Guess what, I’m a researcher. I have published papers.
        famouswaffles 8mo ago
        Okay

cesarvarela 8mo ago

Yeah, but 1800 FIDE players don't make illegal moves, and Gemini does.

famouswaffles 8mo ago
That benchmark methodology isn't great, but regardless, LLMs can be trained to play Chess with a 99.8% legal move rate.
- recursive 8mo ago
  That doesn't exactly sound like strong chess play.
  - dwohnitmok 8mo ago
    It's enough to reliably beat amateur (e.g. maia-1900) chess engines.
dwohnitmok 8mo ago
1800 FIDE players do make illegal moves. I believe they make about one to two orders of magnitude less illegal moves than Gemini 3 does here. IIRC the usual statistic for expert chess play is about 0.02% of expert chess games have an illegal move (I can look that up later if there's interest to be sure), but that is only the ones that made it into the final game notation (and weren't e.g. corrected at the board by an opponent or arbiter). So that should be a lower bound (hence why it could be up to one order lower, although I suspect two orders is still probably closer to the truth). Whether or not we'll see LLMs continue to get a lower error rate to make up for those orders of magnitude remains to be seen (I could see it go either way in the next two years based on the current rate of progress).
- cesarvarela 8mo ago
  A player at that level making an illegal move is either tired, distracted, drunk, etc. An LLM makes it because it does not really "understand" the rules of chess.

runarberg 8mo ago

Wait, I may be missing something here. These benchmarks are gathered by having models play each other, and the second illegal move forfeits the game. This seems like a flawed method as the models who are more prone to illegal moves are going to bump the ratings of the models who are less likely. Additionally, how do we know the model isn’t benchmaxxed to eliminate illegal moves. For example, here is the list of games by Gemini-3-pro-preview. In 44 games it preformed 3 illegal moves (if I counted correctly) but won 5 because opponent forfeits due to illegal moves. https://chessbenchllm.onrender.com/games?page=5&model=gemini-3-pro-preview https://chessbenchllm.onrender.com/games?page=5&model=gemini... I suspect the ratings here may be significantly inflated due to a flaw in the methodology. EDIT: I want to suggest a better methodology here (I am not gonna do it; I really really really don’t care about this technology). Have the LLMs play rated engines and rated humans, the first illegal move forfeits the game (same rules apply to humans).

emp17344 8mo ago
That’s a devastating benchmark design flaw. Sick of these bullshit benchmarks designed solely to hype AI. AI boosters turn around and use them as ammo, despite not understanding them.
- runarberg 8mo ago
  I like this game between grok-4.1-fast and maia-1100 (engine, not LLM). https://chessbenchllm.onrender.com/game/37d0d260-d63b-4e41-9bba-6b25b922493f https://chessbenchllm.onrender.com/game/37d0d260-d63b-4e41-9... This exact game has been played 60 thousand times on lichess. The peace sacrifice Grok performed on move 6 has been played 5 million times on lichess. Every single move Grok made is also the top played move on lichess. This reminds me of Stefan Zweig’s The Royal Game where the protagonist survived Nazi torture by memorizing every game in a chess book his torturers dropped (excellent book btw. and I am aware I just committed Godwin’s law here; also aware of the irony here). The protagonist became “good” at chess, simply by memorizing a lot of games.
  - famouswaffles 8mo ago

overgard 8mo ago

They have literally every chess game in existence to train on, and they can't do better than 1800?

jimbokun 8mo ago
Why do you think they won’t continue to improve?
- overgard 8mo ago
  Because of how LLM's work. I don't know exactly how they're using it for chess, but here's a guess. If you consider the chess game a "conversation" between two opponents, the moves written out would be the context window. So you're asking the LLM, "given these last 30 moves, what's the most likely next move?". Ie, you're giving it a string like "1. e4 e5, 2. Nf3 Nc6, 3. Bb5 a6, 4..?". That's basically what you're doing with LLMs in any context "Here's a set of tokens, what's the most likely continuation?". The problem is, that's the wrong question for a chess move. If you're going with "most likely continuation", that will work great for openings and well-studied move sequences (there are a lot of well studied move sequences!), however, once the game becomes "a brand new game", as chess streamers like to say when there's no longer a game in the database with that set of moves, then "what's the most likely continuation from this position?" is not the right question. Non-LLM AI's have obviously solved chess, so, it doesn't really matter -- I think Chess shows how LLM's lack of a world model as Gary Marcus would say is a problem.