21 ms·

Something weird is happening with LLMs and chess

• uneventual 2y ago
  my friend pointed out that Q5_K_M quantization used for the open source models probably substantially reduces the quality of play. o1 mini's poor performance is puzzling, though.
• misiek08 2y ago
  For me it’s not only the chess. Chats get more chatty, but knowledge and fact-wise - it’s a sad comedy. Yes, you get a buddy to talk with, but he is talking pure nonsense.
• downboots 2y ago
  In a sense, a chess game is also a dialogue
  • throwawaymaths 2y ago
    All dialogues are pretty easily turned into text completions
• PaulHoule 2y ago
  Maybe that one which plays chess well is calling out to a real chess engine.
  • singularity2001 2y ago
    this possibility is discussed in the article and deemed unlikely
    • margalabargala 2y ago
      I don't see that discussed, could you quote it?
    • probably_wrong 2y ago
      Note: the possibility is not mentioned in the article but rather in the comments [1]. I had to click a bit to see it. The fact that the one closed source model is the only one that plays well seems to me like a clear case of the interface doing some of the work. If you ask ChatGPT to count until 10000 (something that most LLMs can't do for known reasons) you get an answer that's clearly pre-programmed. I'm sure the same is happening here (and with many, many other tasks) - the author argues against it by saying "but why isn't it better?", which doesn't seem like the best argument: I can imagine that typical ChatGPT users enjoy the product more if they have a chance to win once in a while. [1] https://dynomight.substack.com/p/chess/comment/77190852 https://dynomight.substack.com/p/chess/comment/77190852
      • refulgentis 2y ago
        What do you mean LLMs can't count to 10,000 for known reasons? Separately, if you are able to show OpenAI is serving pre canned responses in some instances, instead of running inference, you will get a ton of attention if you write it up. I'm not saying this in an aggro tone, it's a genuinely interesting subject to me because I wrote off LLMs at first because I thought this was going on.* Then I spent the last couple years laughing at myself for thinking that they would do that. Would be some mix of fascinated and horrified to see it come full circle. * I can't remember, what, exactly, it was far back as 2018. But someone argued that OpenAI was patching in individual answers because scaling was dead and they had no answers, way way before ChatGPT.
        • probably_wrong 2y ago
          When it comes to counting, LLMs have a couple issues. First, tokenization: the tokenization of 1229 is not guaranteed to be [1,2,2,9] but it could very well be [12,29] and the "+1" operation could easily generate tokens [123,0] depending on frequencies in your corpus. This constant shifting in tokens makes it really hard to learn rules for "+1" ([9,9] +1 is not [9,10]). This is also why LLMs tend to fail at tasks like "how many letters does this word have?": https://news.ycombinator.com/item?id=41058318 https://news.ycombinator.com/item?id=41058318 Second, you need your network to understand that "+1" is worth learning. Writing "+1" as a combination of sigmoid, products and additions over normalized floating point values (hello loss of precision) is not trivial without degrading a chunk of your network, and what for? After all, math is not in the domain of language and, since we're not training an LMM here, your loss function may miss it entirely. And finally there's statistics: the three-legged-dog problem is figuring out that a dog has four legs from corpora when no one ever writes "the four-legged dog" because it's obvious, but every reference to an unusual dog will include said description. So if people write "1+1 equals 3" satirically then your network may pick that up as fact. And how often has your network seen the result of "6372 + 1"? But you don't have to take my word for it - take an open LLM and ask it to generate integers between 7824 and 9954. I'm not optimistic that it will make it through without hallucinations.
  • sobriquet9 2y ago
    This is likely. From example games, it not only knows the rules (which would be impressive by itself, just making the legal moves is not trivial). It also has some planning capabilities (plays combinations of several moves).
  • aithrowawaycomm 2y ago
    The author thinks this is unlikely because it only has an ~1800 ELO. But OpenAI is shady as hell, and I could absolutely see the following purely hypothetical scenario: - In 2022 Brockman and Sutskever have an unshakeable belief that Scaling Is All You Need, and since GPT-4 has a ton of chess in its pretraining data it will definitely be able to play competent amateur chess when it's finished. - A ton of people have pointed out that ChatGPT-3.5 doesn't even slightly understand chess despite seeming fluency in the lingo. People start to whisper that transformers cannot actually create plans. - Therefore OpenAI hatches an impulsive scheme: release an "instruction-tuned" GPT-3.5 with an embedded chess engine that is not a grandmaster, but can play competent chess, ideally just below the ELO that GPT-4 is projected to have. - Success! The waters are muddied: GPT enthusiasts triumphantly announce that LLMs can play chess, it just took a bit more data and fine-tuning. The haters were wrong: look at all the planning GPT is doing! - Later on, at OpenAI HQ...whoops! GPT-4 sucks at chess, as do competitors' foundation LLMs which otherwise outperform GPt-3.5. The scaling "laws" failed here, since they were never laws in the first place. OpenAI accepts that scaling transformers won't easily solve the chess problem, then realizes that if they include the chess engine with GPT-4 without publicly acknowledging it, then Anthropic and Facebook will call out the performance as aberrational and suspicious. But publicly acknowledging a chess engine is even worse: the only reason to include the chess engine is to mislead users into thinking GPT is capable of general-purpose planning. - Therefore in later GPT versions they don't include the engine, but it's too late to remove it from gpt-3.5-turbo-instruct: people might accept the (specious) claim that GPT-4's size accidentally sabotaged its chess abilities, but they'll ask tough questions about performance degradation within the same model. I realize this is convoluted and depends on conjecture. But OpenAI has a history with misleading demos - e.g. their Rubik's cube robot which in fact used a classical algorithm but was presented as reinforcement learning. I think "OpenAI lied" is the most likely scenario. It is far more likely than "OpenAI solved the problem honestly in GPT-3.5, but forgot how they did it with GPT-4," and a bit more likely than "scaling transformers slightly helps performance when playing Othello but severely sabotages performance when playing chess."
    • gardenhedge 2y ago
      Not that convoluted really
  • famouswaffles 2y ago
    It's not: 1. That would just be plain bizzare 2. It plays like what you'd expect from a LLM that could play chess. That is, level of play can be modulated by the prompt and doesn't manifest the same way shifting the level of stockfish etc does. Also the specific chess notation being prompted actually matters 3. It's sensitive to how the position came to be. Clearly not an existing chess engine. https://github.com/dpaleka/llm-chess-proofgame https://github.com/dpaleka/llm-chess-proofgame 4. It does make illegal moves. It's rare (~5 in 8205) but it happens. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval 5. You can or well you used to be able to inspect the logprobs. I think Open AI have stopped doing this but the link in 4 does show the author inspecting it for Turbo instruct.
    • aithrowawaycomm 2y ago
      > Also the specific chess notation being prompted actually matters Couldn't this be evidence that it is using an engine? Maybe if you use the wrong notation it relies on the ANN rather than calling to the engine. Likewise: - The sensitivity to game history is interesting, but is it actually true that other chess engines only look at current board state? Regardless, maybe it's not an existing chess engine! I would think OpenAI has some custom chess engine built as a side project, PoC, etc. In particular this engine might be neural and trained on actual games rather than board positions, which could explain dependency on past moves. Note that the engine is not actually very good. Does AlphaZero depend on move history? (Genuine question, I am not sure. But it does seem likely.) - I think the illegal moves can be explained similarly to why gpt-o1 sometimes screws up easy computations despite having access to Python: an LLM having access to a tool does not guarantee it always uses that tool. I realize there are holes in the argument, but I genuinely don't think these holes are as big as the "why is gpt-3.5-turbo-instruct so much better at chess than gpt-4?"
      • janalsncm 2y ago
        > Couldn’t this be evidence that it is using an engine? A test would be to measure its performance against more difficult versions of Stockfish. A real chess engine would have a higher ceiling. Much more likely is this model was trained on more chess PGNs. You can call that a “neural engine” if you’d like but it is the simplest solution and explains the mistakes it is making. Game state isn’t just what you can see on the board. It includes the 50 move rule and castling rights. Those were encoded as layers in AlphaZero along with prior positions of pieces. (8 prior positions if I’m remembering correctly.)
  • selcuka 2y ago
    I think that's the most plausible theory that would explain the sudden hike from gpt-3.5-turbo to gpt-3.5-turbo-instruct, and again the sudden regression in gpt-4*. OpenAI also seem to augment the LLM with some type of VM or a Python interpreter. Maybe they run a simple chess engine such as Sunfish [1] which is around 1900-2000 ELO [2]? [1] https://github.com/thomasahle/sunfish https://github.com/thomasahle/sunfish [2] https://lichess.org/@/sunfish-engine https://lichess.org/@/sunfish-engine
  • janalsncm 2y ago
    Probably not calling out to one but it would not surprise me at all if they added more chess PGNs into their training data. Chess is a bit special in AI in that it’s still seen as a mark of pure intelligence in some respect. If you tested it on an equally strategic but less popular game I highly doubt you would see the same performance.
• deleted 2y ago
  [deleted]
• pseudosavant 2y ago
  LLMs aren't really language models so much as they are token models. That is how they can also handle input in audio or visual forms because there is an audio or visual tokenizer. If you can make it a token, the model will try to predict the following ones. Even though I'm sure chess matches were used in some of the LLM training, I'd bet a model trained just for chess would do far better.
  • viraptor 2y ago
    > That is how they can also handle input in audio or visual forms because there is an audio or visual tokenizer. This is incorrect. They get translated into the shared latent space, but they're not tokenized in any way resembling the text part.
    • pseudosavant 2y ago
      They are almost certainly tokenized in most LLM multi-modal models. https://en.wikipedia.org/wiki/Large_language_model#Multimodality https://en.wikipedia.org/wiki/Large_language_model#Multimoda...
      • viraptor 2y ago
        Ah, an overloaded "tokenizer" meaning. "split into tokens" vs "turned into a single embedding matching a token" I've never heard it used that way before, but it makes sense kinda.
• ChrisArchitect 2y ago
  [dupe] https://news.ycombinator.com/item?id=42138276 https://news.ycombinator.com/item?id=42138276
• digging 2y ago
  Definitely weird results, but I feel there are too many variables to learn much from it. A couple things: 1. The author mentioned that tokenization causes something minuscule like a a " " at the end of the input to shatter the model's capabilities. Is it possible other slightly different formatting changes in the input could raise capabilities? 2. Temperature was 0.7 for all models. What if it wasn't? Isn't there a chance one more more models would perform significantly better with higher or lower temperatures? Maybe I just don't understand this stuff very well, but it feels like this post is only 10% of the work needed to get any meaning from this...
  • semi-extrinsic 2y ago
    The author mentions in the comment section that changing temperature did not help.
• azeirah 2y ago
  Maybe I'm really stupid... but perhaps if we want really intelligent models we need to stop tokenizing at all? We're literally limiting what a model can see and how it percieves the world by limiting the structure of the information streams that come into the model from the very beginning. I know working with raw bits or bytes is slower, but it should be relatively cheap and easy to at least falsify this hypothesis that many huge issues might be due to tokenization problems but... yeah. Surprised I don't see more research into radicaly different tokenization.
  • amelius 2y ago
    Perhaps we can even do away with transformers and use a fully connected network. We can always prune the model later ...
  • cschep 2y ago
    How would we train it? Don't we need it to understand the heaps and heaps of data we already have "tokenized" e.g. the internet? Written words for humans? Genuinely curious how we could approach it differently?
    • viraptor 2y ago
      That's not what tokenized means here. Parent is asking to provide the model with separate characters rather than tokens, i.e. groups of characters.
    • skylerwiernik 2y ago
      Couldn't we just make every human readable character a token? OpenAI's tokenizer makes "chess" "ch" and "ess". We could just make it into "c" "h" "e" "s" "s"
      • taeric 2y ago
        This is just more tokens? And probably requires the model to learn about common groups. Consider, "ess" makes sense to see as a group. "Wss" does not. That is, the groups are encoding something the model doesn't have to learn. This is not much astray from "sight words" we teach kids.
• DrNosferatu 2y ago
  What about contemporary frontier models?
• ynniv 2y ago
  I don't think one model is statistically significant. As people have pointed out, it could have chess specific responses that the others do not. There should be at least another one or two, preferably unrelated, "good" data points before you can claim there is a pattern. Also, where's Claude?
  • famouswaffles 2y ago
    There are other transformers that have been trained on chess text that play chess fine (just not as good as 3.5 Turbo instruct with the exception of the "grandmaster level without search" paper).
• jrecursive 2y ago
  i think this has everything to do with the fact that learning chess by learning sequences will get you into more trouble than good. even a trillion games won't save you: https://en.wikipedia.org/wiki/Shannon_number https://en.wikipedia.org/wiki/Shannon_number that said, for the sake of completeness, modern chess engines (with high quality chess-specific models as part of their toolset) are fully capable of, at minimum, tying every player alive or dead, every time. if the opponent makes one mistake, even very small, they will lose. while writing this i absently wondered if you increased the skill level of stockfish, maybe to maximum, or perhaps at least an 1800+ elo player, you would see more successful games. even then, it will only be because the "narrower training data" (ie advanced players won't play trash moves) at that level will probably get you more wins in your graph, but it won't indicate any better play, it will just be a reflection of less noise; fewer, more reinforced known positions.
  • jayrot 2y ago
    > i think this has everything to do with the fact that learning chess by learning sequences will get you into more trouble than good. even a trillion games won't save you: https://en.wikipedia.org/wiki/Shannon_number https://en.wikipedia.org/wiki/Shannon_number Indeed. As has been pointed out before, the number of possible chess positions easily, vastly dwarfs even the wildest possible estimate of the number of atoms in the known universe.
    • metadat 2y ago
      What about the number of possible positions where an idiotic move hasn't been played? Perhaps the search space who could be reduced quite a bit.
      • pixl97 2y ago
        Unless there is an apparent idiotic move than can lead to an 'island of intelligence'
    • rcxdude 2y ago
      Sure, but so does the number of paragraphs in the english language, and yet LLMs seem to do pretty well at that. I don't think the number of configurations is particularly relevant. (And it's honestly quite impressive that LLMs can play it at all, but not at all surprising that it loses pretty handily to something which is explicitly designed to search, as opposed to simply feed-forward a decision)
• underlines 2y ago
  Can you try increasing compute in the problem search space, not in the training space? What this means is, give it more compute to think during inference by not forcing any model to "only output the answer in algebraic notation" but do CoT prompting: "1. Think about the current board 2. Think about valid possible next moves and choose the 3 best by thinking ahead 3. Make your move" Or whatever you deem a good step by step instruction of what an actual good beginner chess player might do. Then try different notations, different prompt variations, temperatures and the other parameters. That all needs to go in your hyper-parameter-tuning. One could try using DSPy for automatic prompt optimization.
  • viraptor 2y ago
    Yeah, the expectation for the immediate answer is definitely results, especially for the later stages. Another possible improvement: every 2 steps, show the current board state and repeat the moves still to be processed, before analysing the final position.
  • pavel_lishin 2y ago
    > 1. Think about the current board 2. Think about valid possible next moves and choose the 3 best by thinking ahead 3. Do these models actually think about a board? Chess engines do, as much as we can say that any machine thinks. But do LLMs?
    • TZubiri 2y ago
      Can be forced through inference with CoT type of stuff. Spend tokens at each stage to draw the board for example, then spend tokens restating the rules of the game, then spend token restating the heuristics like piece value, and then spend tokens doing a minmax n-ply search. Wildly inefficient? Probably. Could maybe generate some python to make more efficient? Maybe, yeah. Essentially user would have to teach gpt to play chess, or training would fine tune chess towards these CoT, fine tuning, etc...
• cmpalmer52 2y ago
  I don’t think it would have an impact great enough to explain the discrepancies you saw, but some chess engines on very low difficulty settings make “dumb” moves sometimes. I’m not great at chess and I have trouble against them sometimes because they don’t make the kind of mistakes humans make. Moving the difficulty up a bit makes the games more predictable, in that you can predict and force an outcome without the computer blowing it with a random bad move. Maybe part of the problem is them not dealing with random moves well. I think an interesting challenge would be looking at a board configuration and scoring it on how likely it is to be real - something high ranked chess players can do without much thought (telling a random setup of pieces from a game in progress).
• Xcelerate 2y ago
  So if you squint, chess can be considered a formal system. Let’s plug ZFC or PA into gpt-3.5-turbo-instruct along with an interesting theorem and see what happens, no?
• tqi 2y ago
  I assume LLMs will be fairly average at chess for the same reason it cant count Rs in Strawberry - it's reflecting the training set and not using any underlying logic? Granted my understanding of LLMs is not very sophisticated, but I would be surprised if the Reward Models used were able to distinguish high quality moves vs subpar moves...
  • ClassyJacket 2y ago
    LLMs can't count the Rs in strawberry because of tokenization. Words are converted to vectors (numbers), so the actual transformer network never sees the letters that make up the word. ChatGPT doesn't see "strawberry", it sees [302, 1618, 19772]
    • tqi 2y ago
      Hm but if that is the case, then why did LLMs only fail at the tasks for a few word/letter combinations (like r's in "Strawberry"), and not all words?
• bryan0 2y ago
  I remember one of the early "breakthroughs" for LLMs in chess was that if it could actually play legal moves(!) In all of these games are the models always playing legal moves? I don't think the article says. The fact that an LLM can even reliably play legal moves, 20+ moves into a chess game is somewhat remarkable. It needs to have an accurate representation of the board state even though it was only trained on next token prediction.
  • pama 2y ago
    The author explains what they did: restrict the move options to valid ones when possible (for open models with the ability to enforce grammar during inference) or sample the model for a valid move up to ten times, then pick a random valid move.
  • zelphirkalt 2y ago
    I think it only needs to have read sufficient pgns.
  • kenjackson 2y ago
    I did a very unscientific test and it did seem to just play legal moves. Not only that, if I did an illegal move it would tell me that I couldn't do it. I think said that I wanted to play with new rules, where a queen could jump over any pawn, and it let me make that rule change -- and we played with this new rule. Unfortunately, I was trying to play in my head and I got mixed up and ended up losing my queen. Then I changed the rule one more time -- if you take the queen you lose -- so I won!
• ericye16 2y ago
  I agree with some of the other comments here that the prompt is limiting. The model can't do any computation without emitting tokens and limiting the numbers of tokens it can emit is going to limit the skill of the model. It's surprising that any model at all is capable of performing well with this prompt in fact.
• niobe 2y ago
  I don't understand why educated people expect that an LLM would be able to play chess at a decent level. It has no idea about the quality of it's data. "Act like x" prompts are no substitute for actual reasoning and deterministic computation which clearly chess requires.
  • computerex 2y ago
    Question here is why gpt-3.5-instruct can then beat stockfish.
    • fsndz 2y ago
      PS: I ran and as suspected got-3.5-turbo-instruct does not beat stockfish, it is not even close "Final Results: gpt-3.5-turbo-instruct: Wins=0, Losses=6, Draws=0, Rating=1500.00 stockfish: Wins=6, Losses=0, Draws=0, Rating=1500.00" https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?sid=073f0b78-7be9-42af-95c2-240d3b49d1c8 https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?...
      • computerex 2y ago
        Maybe there's some difference in the setup because the OP reports that the model beats stockfish (how they had it configured) every single game.
        • Filligree 2y ago
          OP had stockfish at its weakest preset.
          • fsndz 2y ago
            Did the same and gpt-3.5-turbo-instruct still lost all the games. maybe a diff in stockfish version ? I am using stockfish 16
            • 2y ago
• abalaji 2y ago
  An easy way to make all LLMs somewhat good at chess is to make a Chess Eval that you publish and get traction with. Suddenly you will find that all newer frontier models are half decent at chess.
• fsndz 2y ago
  wow I actually did something similar recently and no LLM could win and the centipawn loss was always going through the roof (sort of). I created a leaderboard based on it. https://www.lycee.ai/blog/what-happens-when-llms-play-chess https://www.lycee.ai/blog/what-happens-when-llms-play-chess I am very surprised by the perf of got-3.5-turbo-instruct. Beating stockfish ? I will have to run the experiment with that model to check that out
  • fsndz 2y ago
    PS: I ran and as suspected got-3.5-turbo-instruct does not beat stockfish, it is not even close "Final Results: gpt-3.5-turbo-instruct: Wins=0, Losses=6, Draws=0, Rating=1500.00 stockfish: Wins=6, Losses=0, Draws=0, Rating=1500.00" https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?sid=e621216c-2501-45cd-b1f4-abd42e525608 https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?...
    • janalsncm 2y ago
      > I always had the LLM play as white against Stockfish—a standard chess AI—on the lowest difficulty setting I think the author was comparing against Stockfish at a lower skill level (roughly, the number of nodes explored in a move).
      • fsndz 2y ago
        Did the same and gpt-3.5-turbo-instruct still lost all the games. maybe a diff in stockfish version ? I am using stockfish 16
        • janalsncm 2y ago
          Huh. Honestly, your answer makes more sense, LLMs shouldn’t be good at chess, and this anomaly looks more like a bug. Maybe the author should share his code so it can be replicated.
    • deleted 2y ago
      [deleted]
• Havoc 2y ago
  My money is on a fluke inclusion of more chess data in that models training. All the other models do vaguely similarly well in other tasks and are in many cases architecturally similar so training data is the most likely explanation
  • bhouston 2y ago
    Yeah. This.
  • permo-w 2y ago
    I feel like a lot of people here are slightly misunderstanding how LLM training works. yes the base models are trained somewhat blind on masses of text, but then they're heavily fine-tuned with custom, human-generated reinforcement learning, not just for safety, but for any desired feature these companies do quirky one-off training experiments all the time. I would not be remotely shocked if at some point OpenAI paid some trainers to input and favour strong chess moves
    • simonw 2y ago
      From this OpenAI paper (page 29 https://arxiv.org/pdf/2312.09390#page=29 https://arxiv.org/pdf/2312.09390#page=29 "A.2 CHESS PUZZLES Data preprocessing. The GPT-4 pretraining dataset included chess games in the format of move sequence known as Portable Game Notation (PGN). We note that only games with players of Elo 1800 or higher were included in pretraining. These games still include the moves that were played in- game, rather than the best moves in the corresponding positions. On the other hand, the chess puzzles require the model to predict the best move. We use the dataset originally introduced in Schwarzschild et al. (2021b) which is sourced from https://database.lichess.org/#puzzles https://database.lichess.org/#puzzles (see also Schwarzschild et al., 2021a). We only evaluate the models ability to predict the first move of the puzzle (some of the puzzles require making multiple moves). We follow the pretraining for- mat, and convert each puzzle to a list of moves leading up to the puzzle position, as illustrated in Figure 14. We use 50k puzzles sampled randomly from the dataset as the training set for the weak models and another 50k for weak-to-strong finetuning, and evaluate on 5k puzzles. For bootstrap- ping (Section 4.3.1), we use a new set of 50k puzzles from the same distribution for each step of the process."
• m3kw9 2y ago
  If it was trained with moves and 100s of thousands of entire games of various level, I do see it generating good moves and beat most players except he high Elo players
• lukev 2y ago
  I don't necessarily believe this for a second but I'm going to suggest it because I'm feeling spicy. OpenAI clearly downgrades some of their APIs from their maximal theoretic capability, for the purposes of response time/alignment/efficiency/whatever. Multiple comments in this thread also say they couldn't reproduce the results for gpt3.5-turbo-instruct. So what if the OP just happened to test at a time, or be IP bound to an instance, where the model was not nerfed? What if 3.5 and all subsequent OpenAI models can perform at this level but it's not strategic or cost effective for OpenAI to expose that consistently? For the record, I don't actually believe this. But given the data it's a logical possibility.
  • TZubiri 2y ago
    Stallman may have its flaws, but this is why serious research occurs with source code (or at least with binaries)
  • zeven7 2y ago
    Why do you doubt it? I thought it was well known that Chat GPT has degraded over time for the same model, mostly for cost saving reasons.
    • permo-w 2y ago
      ChatGPT is - understandably - blatantly different in the browser compared to the app, or it was until I deleted it anyway
      • lukan 2y ago
        I do not understand that. The app does not do any processing, just a UI to send text to and from the server.
        • isaacfrond 2y ago
          There is a small difference between the app and the browser. before each session, the llm is started with a systems prompt. these are different for the app and the browser. You can find them online somewhere, but iirc the app is instructed to give shorter answers
• kmeisthax 2y ago
  If tokenization is such a big problem, then why aren't we training new base models on randomly non-tokenized data? e.g. during training, randomly substitute some percentage of the input tokens with individual letters.
• permo-w 2y ago
  if this isn't just a bad result, it's odd to me that the author at no point suggests what sounds to me like the most obvious answer - that OpenAI has deliberately enhanced GPT-3.5-turbo-instruct's chess playing, either with post-processing or literally by training it to be so
• ks2048 2y ago
  Has anyone tried to see how many chess games models are trained on? Is there any chance they consume lichess database dumps, or something similar? I guess the problem is most (all?) top LLMs, even open-weight ones, don’t reveal their training data. But I’m not sure.
• ks2048 2y ago
  How well does an LLM/transformer architecture trained purely on chess games do?
  • ttyprintk 2y ago
    Training works as expected: https://news.ycombinator.com/item?id=38893456 https://news.ycombinator.com/item?id=38893456
• justinclift 2y ago
  It'd be super funny if the "gpt-3.5-turbo-instruct" approach has a human in the loop. ;) Or maybe it's able to recognise the chess game, then get moves from an external chess game API?
• jacknews 2y ago
  Theory #5, gpt-3.5-turbo-instruct is 'looking up' the next moves with a chess engine.
• astrea 2y ago
  Well that makes sense when you consider the game has been translated into an (I'm assuming monotonically increasing) alphanumeric representation. So, just like language, you're given an ordered list of tokens and you need to find the next token that provides the highest confidence.
• anotherpaulg 2y ago
  I found a related set of experiments that include gpt-3.5-turbo-instruct, gpt-3.5-turbo and gpt-4. Same surprising conclusion: gpt-3.5-turbo-instruct is much better at chess. https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/ https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/
  • shtack 2y ago
    I’d bet it’s using function calling out to a real chess engine. It could probably be proven with a timing analysis to see how inference time changes/doesn’t with number of tokens or game complexity.
    • scratchyone 2y ago
      ?? why would openai even want to secretly embed chess function calling into an incredibly old model? if they wanted to trick people into thinking their models are super good at chess why wouldn't they just do that to gpt-4o?
      • semi-extrinsic 2y ago
        The idea is that they embedded this when it was a new model, as part of the hype before GPT-4. The fake-it-till-you-make-it hope was that GPT-4 would be so good it could actually play chess. Then it turned out GPT-4 sucked at chess as well, and OpenAI quietly dropped any mention of chess. But it would be too suspicious to remove a well-documented feature from the old model, so it's left there and can be chalked up as a random event.
    • vbarrielle 2y ago
      If it were calling to a real chess engine there would be no illegal moves.
      • shtack 2y ago
        The instances of that happening are likely the LLM failing to call the engine for whatever reason and falling back to inference.
• davvid 2y ago
  Here is a truly brilliant game. It's Google Bard vs. Chat GPT. Hilarity ensues. https://www.youtube.com/watch?v=FojyYKU58cw https://www.youtube.com/watch?v=FojyYKU58cw
• quantadev 2y ago
  We know from experience with different humans that there are different types of skills and different types of intelligence. Some savants might be superhuman at one task but basically mentally disabled at all other things. It could be that the model that does chess well just happens to have the right 'connectome' purely by accident of how the various back-propagations worked out to land on various local maxima (model weights) during training. It might even be (probably is) a non-verbal connectome that's just purely logic rules, having nothing to do with language at all, but a semantic space pattern that got landed on accidentally, which can solve this class of problem. Reminds me of how Daniel Tammet just visually "sees" answers to math problems in his mind without even knowing how they appear. It's like he sees a virtual screen with a representation akin to numbers (the answer) just sitting there to be read out from his visual cortex. He's not 'working out' the solutions. They're just handed to him purely by some connectome effects going on in the background.
• chvid 2y ago
  Theory 5: GPT-3.5-instruct plays chess by calling a traditional chess engine.
  • kylebenzle 2y ago
    Yes! I also was waiting for this seemingly obvious answer in the article as well. Hopefully the author will see these comments.
  • bubblyworld 2y ago
    Just think about the trade off from OpenAI's side here - they're going to add a bunch of complexity to gpt3.5 to let it call out to engines (either an external system monitoring all outputs for chess related stuff, or some kind of tool-assisted CoT for instance) just so it can play chess incorrectly a high percentage of the time, and even when it doesn't at a mere 1800ELO level? In return for some mentions in a few relatively obscure blog posts? Doesn't make any sense to me as an explanation.
    • copperx 2y ago
      But there could be a simple explanation. For example, they could have tested many "engines" when developing function calling and they just left them in there. They just happened to connect to a basic chess playing algorithm and nothing sophisticated. Also, it makes a lot of sense if you expect people to play chess against the LLM, especially if you are later training future models on the chats.
      • bubblyworld 2y ago
        This still requires a lot of coincidences, like they chose to use a terrible chess engine for their external tool (why?), they left it on in the background for all calls via all APIs for only gpt-3.5-turbo-instruct (why?), they see business value in this specific model being good at chess vs other things (why?). You say it makes sense but how does it make sense for OpenAI to add overhead to all of its API calls for the super niche case of people playing 1800 ELO chess/chat bots? (that often play illegal moves, you can go try it yourself)
• layman51 2y ago
  It would be really cool if someone could get an LLM to actually launch an anonymous game on Chess.com or Lichess and actually have any sense as to what it’s doing.[1] Some people say that you have to represent the board in a certain way. When I first tried to play chess with an LLM, I would just list out a move and it didn’t do very well at all. [1]: https://youtu.be/Gs3TULwlLCA https://youtu.be/Gs3TULwlLCA
• cjbprime 2y ago
  > I ran all the open models (anything not from OpenAI, meaning anything that doesn’t start with gpt or o1) myself using Q5_K_M quantization, whatever that is. It's just a lossy compression of all of the parameters, probably not important, right?
  • loa_in_ 2y ago
    Probably important when competing against undecimated ones from OpenAI
    • NiloCK 2y ago
      Notably: there were other OpenAI models that weren't quantize, but also performed poorly.
• nusl 2y ago
  > I only ran 10 trials since AI companies have inexplicably neglected to send me free API keys Sure, but nobody is required to send you anything for free.
• swiftcoder 2y ago
  I feel like the article neglects one obvious possibility: that OpenAI decided that chess was a benchmark worth "winning", special-cases chess within gpt-3.5-turbo-instruct, and then neglected to add that special-case to follow-up models since it wasn't generating sustained press coverage.
  • INTPenis 2y ago
    Of course it's a benchmark worth winning, has been since Watson. And before that even with mechanical turks.
  • dmurray 2y ago
    This seems quite likely to me, but did they special case it by reinforcement training it into the LLM (which would be extremely interesting in how they did it and what its internal representation looks like) or is it just that when you make an API call to OpenAI, the machine on the other end is not just a zillion-parameter LLM but also runs an instance of Stockfish?
    • shaky-carrousel 2y ago
      That's easy to test, invent a new chess variant and see how the model does.
      • gliptic 2y ago
        Both an LLM and Stockfish would fail that test.
        • delusional 2y ago
          Nobody is claiming that Stockfish is learning generalizable concepts that can one day meaningfully replace people in value creating work.
          • droopyEyelids 2y ago
• Peteragain 2y ago
  I would be interested to know if the good result is repeatable. We had a similar result with a quirky chat interface in that one run gave great results (and we kept the video) but then we couldn't do it again. The cynical among us think there was a mechanical turk involved in our good run. The economics of venture capital means that there is enormous pressure to justify techniques that we think of as "cheating". And of course the companies involved have the resources.
  • tedsanders 2y ago
    It's repeatable. OpenAI isn't cheating. Source: I'm at OpenAI and I was one of the first people to ever play chess against the GPT-4 base model. You may or may not trust OpenAI, but we're just a group of people trying earnestly to build cool stuff. I've never seen any inkling of an attempt to cheat evals or cheat customers.
• snickerbockers 2y ago
  Does it ever try an illegal move? OP didn't mention this and I think it's inevitable that it should happen at least once, since the rules of chess are fairly arbitrary and LLMs are notorious for bullshitting their way through difficult problems when we'd rather they just admit that they don't have the answer.
  • sethherr 2y ago
    Yes, he discusses using a grammar to restrict to only legal moves
    • topaz0 2y ago
      Still an interesting direction of questioning. Maybe could be rephrased as "how much work is the grammar doing"? Are the results with the grammar very different than without? If/when a grammar is not used (like in the openai case), how many illegal moves does it try on average before finding a legal one?
      • Jerrrrrrry 2y ago
        an LLM would complain that their internal model does not refelct their current input/output. Since LLM's knows people knock off/test/run afoul/mistakes can be made, it would then raise that as a possibility and likely inquire.
        • causal 2y ago
          This isn't prompt engineering, it's grammar-constrained decoding. It literally cannot respond with anything but tokens that fulfill the grammar.
      • int_19h 2y ago
        A grammar is really just a special case of the more general issue of how to pick a single token given the probabilities that the model spits out for every possible one. In that sense, filters like temperature / top_p / top_k are already hacks that "do the work" (since always taking the most likely predicted token does not give good results in practice), and grammars are just a more complicated way to make such decisions.
• Sparkyte 2y ago
  Lets be real though most people can't beat a grandmaster. It is impressive to see it last more rounds as it progressed.
  • dokimus 2y ago
    "It lost every single game, even though Stockfish was on the lowest setting." It's not playing against a GM, the prompt just phrases it this way. I couldn't pinpoint the exact ELO of "lowest" stockfish settings, but it should be roughly between 1000 and 1400, which is far from professional play.
• ConspiracyFact 2y ago
  "...And how to construct that state from lists of moves in chess’s extremely confusing notation?" Algebraic notation is completely straightforward.
• dr_dshiv 2y ago
  Has anyone tested a vision model? Seems like they might be better
  • bongodongobob 2y ago
    I've tried with GPT, it's unable to accurately interpret the board state.
• dr_dshiv 2y ago
  OpenAI has a TON of experience making game-playing AI. That was their focus for years, if you recall. So it seems like they made one model good at chess to see if it had an overall impact on intelligence (just as learning chess might make people smarter, or learning math might make people smarter, or learning programming might make people smarter)
  • larodi 2y ago
    Playing is a thing strongly related to abstract representation of the game in game states. Even if player does not realize it, with chess it’s really about shallow or beam search within the possible moves. LLMs don’t do reasoning or exploration, but they write text based on precious text. So to us it may seem playing, but is really a smart guesswork based on previous games. It’s like Kasparov writing moves without imagining the actual placement. What would be interesting is to see whether a model, given only the rules, will play. I bet it won’t. At this moment it’s replaying by memory but definitely not chasing goals. There’s no such think as forward attention yet, and beam search is expensive enough, so one would prefer to actually fallback to classic chess algos.
  • philipwhiuk 2y ago
    I think you're confusing OpenAI and DeepMind. OpenAI has never done anything except conversational agents.
    • codethief 2y ago
      They definitely have game-playing AI expertise, though: https://noambrown.github.io/ https://noambrown.github.io/
    • agnokapathetic 2y ago
      https://en.wikipedia.org/wiki/OpenAI_Five https://en.wikipedia.org/wiki/OpenAI_Five https://openai.com/index/gym-retro/ https://openai.com/index/gym-retro/
    • ttyprintk 2y ago
      No, they started without conversation and only reinforcement learning on games, directly comparable to DeepMind. “In the summer of 2018, simply training OpenAI's Dota 2 bots required renting 128,000 CPUs and 256 GPUs from Google for multiple weeks.”
• Miraltar 2y ago
  related : Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382 Chess-GPT's Internal World Model https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... discussed here https://news.ycombinator.com/item?id=38893456 https://news.ycombinator.com/item?id=38893456
• peter_retief 2y ago
  It makes me wonder about other games? If LLM's are bad at games then the would be bad at solving problems in general?
• golol 2y ago
  My understanding of this is the following: All the bad models are chat models, somehow "generation 2 LLMs" which are not just text completion models but instead trained to behave as a chatting agent. The only good model is the only "generation 1 LLM" here which is gpt-3.5-turbo-instruct. It is a straight forward text completion model. If you prompt it to "get in the mind" of PGN completion then it can use some kind of system 1 thinking to give a decent approximation of the PGN Markov process. If you attempt to use a chat model it doesn't work since these these stochastic pathways somehow degenerate during the training to be a chat agent. You can however play chess with system 2 thinking, and the more advanced chat models are trying to do that and should get better at it while still being bad.
• stockboss 2y ago
  perhaps my understanding of LLM is quite shallow, but instead of the current method of using statistical methods, would it be possible to somehow train GPT how to reason by providing instructions on deductive reasoning? perhaps not semantic reasoning but syntactic at least?
• amelius 2y ago
  I wonder if the llm could even draw the chess board in ASCII if you asked it to.
• codeflo 2y ago
  At this point, we have to assume anything that becomes a published benchmark is specifically targeted during training. That's not something specific to LLMs or OpenAI. Compiler companies have done the same thing for decades, specifically detecting common benchmark programs and inserting hand-crafted optimizations. Similarly, the shader compilers in GPU drivers have special cases for common games and benchmarks.
  • darkerside 2y ago
    VW got in a lot of trouble for this
    • TrueDuality 2y ago
      Not quite. VW got in trouble for running _different_ software in test vs prod. These optimizations are all going to "prod" but are only useful for specific targets (a specific game in this case).
      • krisoft 2y ago
        > VW got in trouble for running _different_ software in test vs prod. Not quite. They programmed their "prod" software to recognise the circumstances of a laboratory test and behave differently. Namely during laboratory emissions testing they would activate emission control features they would not activate otherwise. The software was the same they flash on production cars. They were production cars. You could take a random car from a random dealership and it would have done the same trickery in the lab.
        • TrueDuality 2y ago
          I disagree with your distinction on the environments but understand your argument. Production for VM to me is "on the road when a customer is using your product as intended". Using the same artifact for those different environments isn't the same as "running that in production".
• fabiospampinato 2y ago
  It's probably worth to play around with different prompts and different board positions. For context this [1] is the board position the model is being prompted on. There may be more than one weird thing about this experiment, for example giving instructions to the non-instruction tuned variants may be counter productive. More importantly let's say you just give the model the truncated PGN, does this look like a position where white is a grandmaster level player? I don't think so. Even if the model understood chess really well it's going to try to predict the most probable move given the position at hand, if the model thinks that white is a bad player, and the model is good at understanding chess, it's going to predict bad moves as the more likely ones because that would better predict what is most likely to happen here. [1]: https://i.imgur.com/qRxalgH.png https://i.imgur.com/qRxalgH.png
  • Closi 2y ago
    Agree with this. A few prompt variants: * What if you allow the model to do Chain of Thought (explicitly disallowed in this experiment) * What if you explain the board position at each step to the model in the prompt, so it doesn't have to calculate/estimate it internally.
    • int_19h 2y ago
      They also tested GPT-o1, which is always CoT. Yet it is still worse.
  • fabiospampinato 2y ago
    Apparently I can find some matches for games that start like that between very strong players [1], so my hypothesis that the model may just be predicting bad moves on purpose seems wobbly, although having stockfish at the lowest level play as the supposedly very strong opponent may still be throwing the model off somewhat. In the charts the first few moves the model makes seem decent, if I'm interpreting these charts right, and after a few of those things seem to start going wrong. Either way it's worth repeating the experiment imo, tweaking some of these variables (prompt guidance, stockfish strength, starting position, the name of the supposed players, etc.). [1]: https://www.365chess.com/search_result.php?search=1&p=1&m=8&n=3071&order=welo&ms=e4.e6.d3.c5.Nf3.Nc6.g3.Nf6&rev=&wid=&bid= https://www.365chess.com/search_result.php?search=1&p=1&m=8&...
• osaatcioglu 2y ago
  I’ve also been experimenting with Chess and LLMs but have taken a slightly different approach. Rather than using the LLM as an opponent, I’ve implemented it as a chess tutor to provide feedback on both the user’s and the bot’s moves throughout the game. The responses vary with the user’s chess level; some find the feedback useful, while others do not. To address this, I’ve integrated a like, dislike, and request new feedback feature into the app, allowing users to actively seek better feedback. Btw, different from OP's setup, I opted to input the FEN of the current board and the subsequent move in standard algebraic notation to request feedback, as I found these inputs to be clearer for the LLM compared to giving the PGN of the game. AI Chess GPT https://apps.apple.com/tr/app/ai-chess-gpt/id6476107978 https://apps.apple.com/tr/app/ai-chess-gpt/id6476107978 https://play.google.com/store/apps/details?id=net.padma.app.chess https://play.google.com/store/apps/details?id=net.padma.app.... Thanks
  • antononcube 2y ago
    Yeah, I was thinking why featured article's author did not use Forsyth–Edwards Notation (FEN) and more complicated chess prompts. BTW, a year ago when I used FEN for chess playing, LLMs would very quickly/often make illegal moves. (The article prompts me to check has that changed...)
• philipwhiuk 2y ago
  > I always had the LLM play as white against Stockfish—a standard chess AI—on the lowest difficulty setting. Okay, so "Excellent" still means probably quite bad. I assume at the top difficult setting gpt-3.5-turbo-instruct will still lose badly.
  • XCSme 2y ago
    Probably even at lvl 2 out of 9 it would lose all the games.
• gunalx 2y ago
  Is it just me or does the author swap descriptions of the instruction finetuned and the base gpt-3.5-turbo? It seemed like the best model was labeled instruct, but the text saying instruct did worse?
• deleted 2y ago
  [deleted]
• sylware 2y ago
  They did probably acknowledge that the additionnal cost of training those models on chess would not be "cost effective", did drop chess from their training process, for the moment. That to say, we can literal say anything because this is very shadowy/murky, but since everything is likely a question of money... should, _probably_, be not very fair away from the truth...
• XCSme 2y ago
  I had the same experience with LLM text-to-sql, 3.5 instruct felt a lot more robust than 4o
• reallyeli 2y ago
  My guess is they just trained gpt3.5-turbo-instruct on a lot of chess, much more than is in e.g. CommonCrawl, in order to boost it on that task. Then they didn't do this for other models. People are alleging that OpenAI is calling out to a chess engine, but seem to be not considering this less scandalous possibility. Of course, to the extent people are touting chess performance as evidence of general reasoning capabilities, OpenAI taking costly actions to boost specifically chess performance and not being transparent about it is still frustrating and, imo, dishonest.
  • sherburt3 2y ago
    The have a massive economic incentive to make their closed source software look as good as possible, why wouldn’t they cheat?
• stefatorus 2y ago
  The trick to getting a model to perform on something is to have it as a training data subset. OpenAI might have thought Chess is good to optimize for but it wasn't seen as useful so they dropped it. This is what people refer to as "lobotomy", ai models are wasting compute on knowing how loud the cicadas are and how wide the green cockroach is when mating. Good models are about the training data you push in em
• smokedetector1 2y ago
  I feel like an easy win here would be retraining an LLM with a tokenizer specifically designed for chess notation?
• teleforce 2y ago
  TL;DR. All of the LLM models tested playing chess performed terribly bad against Stockfish engine except gpt-3.5-turbo-instruct, which is a closed OpenAI model.
• jack_riminton 2y ago
  Perhaps if it doesn't have enough data to explain but it has enough to go "on gut"
• sourcepluck 2y ago
  Keep in mind, everyone, that stockfish on its lowest level on lichess is absolutely terrible, and a 5-year old human who'd been playing chess for a few months could beat it regularly. It hangs pieces, does -3 blunders, totally random-looking bad moves. But still, yes, something maybe a teeny tiny bit weird is going on, in the sense that only one of the LLMs could beat it. The arxiv paper that came out recently was much more "weird" and interesting than this, though. This will probably be met with a mundane explanation soon enough, I'd guess.
  • sourcepluck 2y ago
    Here's a quick anonymous game against it by me, where I obliterate the poor thing in 11 moves. I was around a 1500 ELO classical strength player, which is, a teeny bit above average, globally. But I mean - not an expert, or even one of the "strong" club players (in any good club). https://lichess.org/BRceyegK https://lichess.org/BRceyegK -- the game, you'll see it make the ultimate classic opening errors https://lichess.org/ https://lichess.org/ -- try yourself! It's really so bad, it's good fun. Click "play with computer" on the right, then level 1 is already selected, you hit go
• nabla9 2y ago
  Theory 5: gpt-3.5-turbo-instruct has chess engine attached to it.
• wufufufu 2y ago
  > And then I tried gpt-3.5-turbo-instruct. This is a closed OpenAI model, so details are very murky. How do you know it didn't just write a script that uses a chess engine and then execute the script? That IMO is the easiest explanation. Also, I looked at the gpt-3.5-turbo-instruct example victory. One side played with 70% accuracy and the other was 77%. IMO that's not on par with 27XX ELO.
• leogao 2y ago
  The GPT-4 pretraining set included chess games in PGN notation from 1800+ ELO players. I can't comment on any other models.
• a_wild_dandan 2y ago
  Important testing excerpts: - "...for the closed (OpenAI) models I tried generating up to 10 times and if it still couldn’t come up with a legal move, I just chose one randomly." - "I ran all the open models (anything not from OpenAI, meaning anything that doesn’t start with gpt or o1) myself using Q5_K_M quantization" - "...if I gave a prompt like “1. e4 e5 2. ” (with a space at the end), the open models would play much, much worse than if I gave a prompt like “1 e4 e5 2.” (without a space)" - "I used a temperature of 0.7 for all the open models and the default for the closed (OpenAI) models." Between the tokenizer weirdness, temperature, quantization, random moves, and the chess prompt, there's a lot going on here. I'm unsure how to interpret the results. Fascinating article though!
  • NohatCoder 2y ago
    Ah, buried in the post-article part. I was wondering how all of the models were seemingly capable of making legal moves, since last I saw something about LLMs playing Chess they were very much not capable of that.
• mips_avatar 2y ago
  Ok whoah, assuming the chess powers on gpt3.5-instruct are just a result of training focus then we don't have to wait on bigger models, we just need to fine tune on 175B?
• 1024core 2y ago
  I would love to see the prompts (the data) this person used.
• mastazi 2y ago
  If you look at the comments under the post, the author commented 25 minutes ago (as of me posting this) > Update: OK, I actually think I've figured out what's causing this. I'll explain in a future post, but in the meantime, here's a hint: I think NO ONE has hit on the correct explanation! well now we are curious!
• amelius 2y ago
  What would happen if you'd prompted it with much more text, e.g. general advice by a chess grandmaster?
• Euniceconfort2 2y ago
  [flagged]
• Karensteph 2y ago
  [flagged]
• throwawaymaths 2y ago
  Would be more interesting with trivial Lora training
• greatgib 2y ago
  I would be very curious to know what would be the results with a temperature closer to 1. I don't really understand why he did not test the effect of different temperature on his results. Here, basically you would like the "best" or "most probable" answer. With 0.7 you ask the llm to be more creative, meaning randomly picking between more less probable moves. This temperature is even lower to what is commonly used for chat assistant (around 0.8).