10 ms·
This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to
by carodgers 11d ago
This April 2026 paper is a fun and related read.
https://arxiv.org/html/2509.24239v4 https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
- threethirtytwo 11d agoThe story isn't so clear cut. The caveat is: It depends on the task. Are there reams of chess moves that the model can train off of? No. Are there reams of math papers the model can train off of? Yes.
- deleted 11d ago[deleted]
- iwontberude 11d ago[dead]
- keephnacct 11d ago[flagged]
- tjwebbnorfolk 11d ago> Are there reams of chess moves that the model can train off of? No. This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
- XenophileJKO 11d agoIt is even worse.. This is a classical reinforcement problem where data generation is easy because the rule set is pre-defined. So you really don't even need any data to start with (but would help).
- manquer 11d agoThere are more possible game combinations than atoms in the universe, even those generation of valid game states are as you say pre-defined. that is why models cannot go this route and therefore are poor at chess
- wat10000 11d agoIsn’t this exactly how AlphaZero was trained? The rules are known and well defined so the training process can generate games without any outside data. The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
- threethirtytwo 10d agoLet me make my statement more clear with a correction: Was there reams of chess moves that the model trained off of? No.
- vmg12 11d ago> The caveat is: It depends on the task. I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models. The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for. Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
- freejazz 11d ago>Are there reams of chess moves that the model can train off of? No. For real??
- threethirtytwo 10d agoI meant if there are reams of chess moves the model was trained off of.
- freejazz 9d agoYeah, it's not like there's any literature about Chess in the corpus of these models!
- threethirtytwo 8d agoThere’s literature. But I don’t think there’s reams of chess games. With literature the LLM can understand strategy but chess needs intuition and you need tokenized games for that. LLMs have less of that.
- FuckButtons 11d agoThere’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
- thelaxiankey 10d agothere are far more reams of chess moves than there are math papers. Lichess is pretty open... But hey, they're actually good at chess if you prompt correctly so.... https://dynomight.net/more-chess/ https://dynomight.net/more-chess/
- wat10000 11d agoI wonder how current models would fare. The ones they tested are fairly old now.
- joefourier 11d ago> current frontier models > Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1 The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
- deleted 11d ago[deleted]
- sigmoid10 11d agoThe actual current frontier plays somewhere around GM level. https://chessbench-ai.github.io/#leaderboard https://chessbench-ai.github.io/#leaderboard It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
- htrp 11d agomore like you lose intelligence in chess by maxing for coding... hence knocking back the claims of emergent intelligence
- csande17 11d agoEven if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.
- MichaelNolan 11d agoI wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
- consensus1 11d agoThis isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
- nefarious_ends 11d agoThanks for saying this, feels like everyone has gone insane over this stuff.
- what 11d agoHumans don’t code a $game engine to play $game, they can just play it. It seems like you are the one that has gone insane.
- hackinthebochs 11d agoAnd how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.
- sph 10d agoNo, learning is definitely not a sign of super intelligence. I know words don’t mean anything anymore, but that is simply general intelligence, despite the claims we have reached this milestone.
- hackinthebochs 10d agoNo, but superhuman capabilities derived from ordinary learning is, which is what the parent comment described. Why is that not obvious?
- 10d ago
- deleted 11d ago[deleted]
- famouswaffles 11d agoFrontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves. Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
- bigstrat2003 11d ago> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
- skydhash 11d agoPretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
- willmarch 11d agoIf you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn). Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
- WhitneyLand 11d ago1. It’s hard to trust a 2026 paper that’s showing results for such old models. 2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks. 3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
- manquer 11d ago> People who are good at it rely more on experience and deep domain expertise People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range. A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
- svachalek 11d ago1100 at online speed chess or something, could be. I'm not that deep in the chess world but everyone I know that can make 1100 in official rating can name a dozen openings and most of the known tactics, and is pretty good at applying at least one opening.
- what 11d ago> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly. Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
- matteoraso 11d agoI don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.
- gizmodo59 11d agowhy cant models make a tool call to stockfish? its like saying model can't execute python for complex math calculations
- 1dom 10d agoThe last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do. > "current frontier models need laborious oversight and guardrails on even the simplest tasks" I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now. What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs? Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
- stinkbeetle 10d ago> Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO. It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly. > The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct. I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.
- contubernio 10d agoThey should just make them read an old chess book like Lasker that gives low level heuristics
- tossandthrow 10d agoLlm systems are not really build for adhering to a grammar (other than "a string og tokens"). It is also not clear whether the llm adhering to a grammar is necessary for intelligent agents. Certainly,a harness can easily correct for it.
- vkazanov 10d agoBy the promise of it, llms should be able to both adhere to grammars, or go free form where necessary. I mean, doing math is supposed to be strict but in practice it's a somewhat educated random walk in the space of correct lean theorems. Harnesses do correct things, sure.
- tossandthrow 10d agoYou are right. I am imprecise. Languages allow a certain flexibility in their grammars - you can read a sentence without that adhering it exactly to the grammar. Games and programming languages (including lean) does not allow this flexibility. A very intelligent person would likely also reason in terms of probably outcomes before correcting a statement to adhering entirely to the grammar. Certainly it must be like that, otherwise reviews in math was rendered moot. Do we blame research mathematicians for not adhering to the grammar?
- wodenokoto 10d agoIt seems absolutely crazy to me to expect an LLM to code a solution to a problem while also not expecting it to be able to adhere to a grammar.
- tossandthrow 10d agoWhy? You might never have tried to program before, so I don't blame it on you. But most programmers, even experienced ones, see grammar and type errors regularly.
- Gregkion 10d agoHow much support do we as humans need to get rules right? I'm an expert in my field, read my comments, my gramma is shit.
- Auracle 10d agoThe fact that they can play chess at all despite having no specific training for it blows my mind, and the fact it doesn’t do the same for many others shows just how far they’ve come and how fast.
- bigstrat2003 10d agoIt doesn't blow anyone's mind because it hasn't been impressive for a computer to play chess for 40 years. "We made something worse than existing solutions by using a new technique" is not an impressive feat.
- Auracle 8d agoIt's literally generalized intelligence. Yeah, I had a shitty chessboard back in the 90's that was specifically programmed to play chess. That same board couldn't tell me the best medication to use to treat a certain condition or how best to modify a model in Blender.
- kbau 10d agoI suspect (in a probably ignorant fashion) that this is because learning process has been reading a lot of algebraic chess notation (such as "1. e4 e5 2. Nf3 f6 3. Nxf6 gxf6 4. Qh5! +-") then, to play, generating more of it without considering the rules of the game. This is exactly how it's always felt to me when playing chess against LLMs. Sure, "1. e4 e5 2. Nf3 Nc3" looks innocent to somebody simply learning the syntax of algebraic notation, but that Nc3 by black is an illegal move. An LLM is the wrong approach for playing chess.
- JohnKemeny 10d agoAre you saying that modern LLMs cannot play chess now, or that LLMs (GPT architecture) cannot be trained to play chess well? Or are you saying that neural networks in general cannot (practically) be trained to be an above-average chess player? Or are you saying that it depends on the input? Would it be better if they were given a picture/drawing/ascii art of the board? If so, surely they can produce it at will?
- kbau 10d agoNeural Nets can be trained to play chess very well and have been doing so for a long time (see Stockfish and Leela as some of the most popular/strongest ones - top GMs have no chance against them), but these are dedicated models, where the game rules are encoded in the learning process, as opposed to large language models which are natural language processing models. Technically you can give an LLM a lot of chess books and games and it will be able to spit out chess notation. Put a webapp on top that renders text moves to the board and it looks like it's playing chess. But it isn't really.
- bitexploder 10d agohttps://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room I think about this once in a while. At some point if it does the thing almost perfectly is it still not doing the thing?
- 10d ago
- ricky54 10d agoIf you give the same task to an exceptionally intelligent human, who does not play chess and has only heard about it in passing, then they would be beaten by every child who has looked at the rules for more than 10 minutes. What kind of intelligence is "playing <____> but we don't tell you the rules" supposed to test?
- bulder 10d ago...but that's not what the models are. You can interrogate them on the rules of chess, and they'll (statistically likely) give you a decent breakdown of the rules. Evidently the rules are in their training material, they just fail to apply them in the manner of an intelligent system for some reason or another.
- lynx97 10d agoWell, yes, PGN files have structure... But still, playing Chess with an LLM is so weird that I impulsively question the sanity of people attempting to do so. Do some people really believe training on TWIC PGNs would make an LLM a good chess player?
- killerstorm 10d agoThis is an absolute nonsense. Any frontier model can implement chess program from scratch - modeling the board, checking legality, etc. If you asked e.g. GPT-6 to get good at chess and gave it a computer, it will get good at chess. That's an actual strategic skill. Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for. This is also true for humans - people who don't have years of chess training can't really tell which moves are legal given an algebraic notation transcript. These people might have good strategic skills in different areas. Chess is just a very, very specific skill
- roenxi 10d agoIt's an interesting puzzle, isn't it. On the one hand, the AIs are no good at playing Chess. However, on the other hand, if you ask an AI to win a game of chess it has all the tools on hand to compete at the same level as Stockfish - it can re-implement an engine and even probably has a GPU on hand to train its own neural nets. So should we say that the AI can play chess well, or that it cannot?
- tired-turtle 10d agoIs it, though? If you design and build a winning F1 race car, did you also win the race? Recent discourse around AI seems to conflate the semantics of winning: 1. you contributed to the win vs 2. you yourself were the winning driver.
- roenxi 10d agoBut the human would have to get extra hardware to do that. The AI isn't bringing in any resources it doesn't already have access to.
- kbau 10d agoI can compile stockfish from source and use it to beat other kids in my class in chess. Behold, I am a chess genius.
- glitchc 10d agoWhy not ask it to implement a chess engine first, and then use that to play against you? Does the LLM need to learn to play chess if it can build a chess engine to play for it instead?
- lionkor 10d agoWith that approach, the benchmark falls apart. Of course it can write a chess engine, because it learned on lots of stolen source code of chess engines. This has nothing to do with the LLM's ability to reason. Writing a well understood engine for a super popular problem does not count as reasoning about the problem.
- glitchc 10d ago> Writing a well understood engine for a super popular problem does not count as reasoning about the problem. Doesn't writing the engine imply understanding about the problem domain? Tool use is a widely accepted measure of intelligence.
- lionkor 10d agoIn most humans, yes, because we are terrible at memorizing millions of codebases. For LLMs, we need to apply our understanding of them before making statements like that. An LLM can "memorize", and has "memorized"/been trained on tens of thousands of chess engines. Writing a chess engine, or even deriving a chess engine from the rules alone, does not constitute a deep understanding of, and more importantly, the ability to apply, the rules, at all. When humans do this, they inadvertently learn something, too, but when an LLM reproduces or derives and implementation of a chess engine, it in no way implies that the LLM can follow the rules in its own "train of thought" and consistently apply the rules in its "head". Let's say you want to evaluate my algebra skills. You make me solve some algebra challenges. If I then whip out a computer and write a calculator, or take some sticks and stones and take a couple hours to build an abacus, and then solve the algebraic challenges, this would not constitute a good solution, and would defeat the entire point of the test. If, instead, I do the algebra in my head or on paper, it might seem like there's no difference, but you can derive all sorts of information from that. For example, you could time it, check for recurring errors I make, for interesting mistakes like mistaking 7 and 1 for one another due to bad hand-writing, etc. If that was the goal, then me writing a calculator or crafting an abacus defeats the point of the test. Yes, me writing a calculator shows that I'm intelligent, and I understand the algebraic rules, but if the test is about applying the rules, I have not passed. In the very same way, an LLM writing a chess engine to solve a chess benchmark that is all about LLM's reasoning capability is complete bogus and defeats the entire point.
- dzonga 10d agopeople on the ground know that small models are enough, since llms are good at directed work (i.e handholding) not the let loose go wild that the labs try to hype on. the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.
- MattCruikshank 10d agoWhat happens when you ask those same frontier models to write a chess-playing program? I feel like, this is a huge stumbling block that many people have. They'll give a model their data, and ask it questions. I vastly prefer letting the model understand the schema, and then writing functions or programs to answer those questions. I feel like I get way, way better answers. I can have it write unit tests for those functions. I can fuzz test those functions. I can look for data that doesn't fit the schema. I can process new data way faster (and with fewer tokens). I can repeatably get the same answers from the same inputs. I can check the code into a git repo and track changes to it over time. I can share the code with other people. I can review the code. I can improve the speed of the code and get the same answers. I can review the error accumulation, and improve it. I can decide how to handle anomalies, and encode those answers. It's really neat to see what a frontier model can do itself. No doubt. But "play chess by hand" is a frankly awful metric. It's kind of like asking someone to take a cube root of some arbitrary decimal, in their head, with no scratch paper.
- cbolton 10d agoIt's a great test of cognitive abilities. There are many claims that current LLMs surpass humans in cognitive abilities so it's noteworthy that they underperform on that test. Letting the model execute a chess program (that it wrote) would make sense if you're measuring its economic potential, but for cognition that would be cheating just like if you let a human run a chess program. The fact that the human would have a much harder time writing a useful program is irrelevant.
- MattCruikshank 10d ago> The fact that the human would have a much harder time writing a useful program is irrelevant. Why? There's a box. You give it a problem, and it comes up with a solution. Why does it matter to you if the box is strictly a LLM, or if the LLM can write code that it executes? Even neater if the box is self-contained with a local model. You provide electricity, and it comes up with solutions. Why does it matter if it can do chess "in its head", or if it has to use scratch paper?
- tzone 10d agoIt all depends on what prompt you use though. You can just tell all current frontier models to write a chess engine first, and then play a game of chess against you using that engine. It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun). On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer". It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff. Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend an infinite amount of money.
- causal 10d agoYeah, thread full of cope. "Well if you remove the human's legs it's actually quite bad at marathons" arguments.
- empath75 10d agoI want you to consider how relevant this is in any practical sense. First -- most _people_ cannot do this, without having a physical board in front of them. Second -- Claude Code is perfectly capable of downloading and running stockfish. People focus too much on LLMs by themselves as the entity of concern instead of the entire harness and all of it's capabilities together.
- Capricorn2481 10d agoBecause they are obviously testing for general intelligence. If you want a thread about how cool the harness is, that's down the street. We don't really consider humans downloading stockfish to beat people at chess as noteworthy endeavors.
- thelaxiankey 10d agoThese "researchers" are less informed on LLM chess than random internet bloggers. The situation is much more interesting https://dynomight.net/chess/ https://dynomight.net/chess/
- thelaxiankey 10d agoForgot to add the amazing follow up https://dynomight.net/more-chess/ https://dynomight.net/more-chess/
- topaz0 10d agoFrom your linked post: "LLMs sometimes struggle to give legal moves. In these experiments, I try 10 times and if there’s still no legal move, I just pick one at random." Which sounds a lot like what that paper was about
- m3at 10d agoYes "oversight and guardrails" are still needed, but even that is becoming easier to build, and imo already no longer in the "laborious" category. Even far from the frontier, you can tune a 0.2B LLM into a decent 2000 Elo player as a weekend project: https://x.com/maximelabonne/status/2100137121264828901 https://x.com/maximelabonne/status/2100137121264828901