24 ms·

OK, I can partly explain the LLM chess weirdness now

• atemerev 2y ago
  Ah, half of the commentariat still think that “LLMs can’t reason”. Even if they have enough state space for reasoning, and clearly demonstrate that.
  • brookst 2y ago
    But it's not real reasoning because it is just outputting likely next tokens that are identical to what we'd expect with reasoning. /s
  • lottin 2y ago
    "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra
  • sourcepluck 2y ago
    Most people, as far as I'm aware, don't have an issue with the idea that LLMs are producing behaviour which gives the appearance of reasoning as far as we understand it today. Which essentially means, it makes sentences that are gramatical, responsive and contextual based on what you said (quite often). It's at least pretty cool that we've got machines to do that, most people seem to think. The issue is that there might be more to reason than appearing to reason. We just don't know. I'm not sure how it's apparently so unknown or unappreciated by people in the computer world, but there are major unresolved questions in science and philosophy around things like thinking, reasoning, language, consciousness, and the mind. No amount of techno-optimism can change this fact. The issue is we have not gotten further than more or less educated guesses as to what those words mean. LLMs bring that interesting fact to light, even providing humanity with a wonderful nudge to keep grappling with these unsolved questions, and perhaps make some progress. To be clear, they certainly are sometimes passably good when it comes to summarising selectively and responsively the terabytes and terabytes of data they've been trained on, don't get me wrong, and I am enjoying that new thing in the world. And if you want to define reason like that, feel free.
    • atemerev 2y ago
      LLMs can _play chess_. With the game positions previously unseen. How’s that not actual logical reasoning?
      • sourcepluck 2y ago
        I guess you don't follow TCEC, or computer chess generally[0]. Chess engines have been _playing chess_ at superhuman levels using neural networks for years now, it was a revolution in the space. AlphaZero, Lc0, Stockfish NNUE. I don't recall yards of commentary arguing that they were reasoning. Look, you can put as many underscores as you like, the question of whether these machines are really reasoning or emulating reason is not a solved problem. We don't know what reasoning is! We don't know if we are really reasoning, because we have major unresolved questions regarding the mind and consciousness[1]. These may not be intractable problems either, there's reason for hope. In particular, studying brains with more precision is obviously exciting there. More computational experiments, including the recent explosion in LLM research, is also great. Still, reflexively believing in the computational theory of the mind[2] without engaging in the actual difficulty of those questions, though commonplace, is not reasonable. [0] Jozarov on YT has great commentary of top engine games, worth checking out. [1] https://plato.stanford.edu/entries/consciousness/ https://plato.stanford.edu/entries/consciousness/ [2] https://plato.stanford.edu/entries/computational-mind/ https://plato.stanford.edu/entries/computational-mind/
        • atemerev 2y ago
          I am not implying that LLMs are conscious or something. Just that they can reason, i.e. draw logical conclusions from observations (or, in their case, textual inputs), and make generalizations. This is a much weaker requirement. Chess engines can reason about chess (they can even explain their reasoning). LLMs can reason about many other things, with varied efficiency. What everyone is currently trying to build is something like AlphaZero (adversarial self-improvement for superhuman performance) with the state space of LLMs (general enough to be useful for most tasks). When we’ll have this, we’ll have AGI.
    • 2y ago
• amrrs 2y ago
  >Theory 1: Large enough base models are good at chess, but this doesn’t persist through instruction tuning to chat models. I lean mostly towards this and also the chess notations - not sure if it might get chopped during tokenization unless it's very precisely processed. It's like designing an LLM just for predicting protein sequence because the sequencing matters. The base data might have it but i don't think that's the intention for it to continue.
  • com2kid 2y ago
    This makes me wonder what scenarios would be unlocked if OpenAI gave access to gpt4-instruct. I wonder if they avoid that due to the potential for negative press from the outputs of a more "raw" model.
• OutOfHere 2y ago
  [flagged]
  • synarchefriend 2y ago
    The model was trained on games in PGN notation. It would be shocking if it found ASCII art easier to understand than what it was actually trained on.
    • OutOfHere 2y ago
      Well, clearly you're not interested in experimentation, only in assumptions.
      • shitpostbot 2y ago
        [dead]
      • daveguy 2y ago
        How does stating the outcome you expect imply you are not interested in experimentation? Hypothesis formation is the very first step in experimentation.
      • danielmarkbruce 2y ago
        Most people who understand LLMs and how they are trained would be shocked. In practice, that's an objectively true statement.
      • BeetleB 2y ago
        Please, please show us your experiments.
        • OutOfHere 2y ago
          I am not the one writing and posting useless articles, even harmful articles, also distorting the understanding of LLMs. Ask the ones who do to perform better experiments.
• tromp 2y ago
  > For one, gpt-3.5-turbo-instruct rarely suggests illegal moves, even in the late game. This requires “understanding” chess. Here's one way to test whether it really understands chess. Make it play the next move in 1000 random legal positions (in which no side is checkmated yet). Such positions can be generated using the ChessPositionRanking project at [1]. Does it still rarely suggest illegal moves in these totally weird positions, that will be completely unlike any it would have seen in training (and in which the legal move choice is often highly restricted) ? While good for testing legality of next moves, these positions are not so useful for distinguishing their quality, since usually one side already has an overwhelming advantage. [1] https://github.com/tromp/ChessPositionRanking https://github.com/tromp/ChessPositionRanking
  • deleted 2y ago
    [deleted]
  • BurningFrog 2y ago
    Not that I understand the internals of current AI tech, but... I'd expect that an AI that has seen billions of chess positions, and the moves played in them, can figure out the rules for legal moves without being told?
    • rscho 2y ago
      Statistical 'AI' doesn't 'understand' anything, strictly speaking. It predicts a move with high probability, which could be legal or illegal.
      • griomnib 2y ago
        Likewise with LLM you don’t know if it is truly in the “chess” branch of the statistical distribution or it is picking up something else entirely, like some arcane overlap of tokens. So much of the training data (eg common crawl, pile, Reddit) is dogshit, so it generates reheated dogshit.
        • Helonomoto 2y ago
• viraptor 2y ago
  I'm glad he improved the promoting, but he's still leaving out two likely huge improvements. 1. Explain the current board position and the plan going forwards, before proposing a move. This lets the model actually think more, kind of like o1, but here it would guarantee a more focused processing. 2. Actually draw the ascii board for each step. Hopefully producing more valid moves since board + move is easier to reliably process than 20×move.
  • unoti 2y ago
    I came here to basically say the same thing. The improvements the OP saw by asking it to repeat all the moves so far gives the LLM more time and space to think. I have this hypothesis giving it more time and space to think in other ways could improve performance even more, something like showing the current board position and asking it to perform an analysis of the position, list key challenges and strengths, asking it for a list of strategies possible from here, then asking it to select a strategy amongst the listed strategies, then asking it for its move. In general, asking it to really think rather than blurt out a move. The examples would be key here. These ideas were proven to work very well in the ReAct paper (and by extension, the CoT Chain of Thought paper). Could also extend this by asking it to do this N times and stop when we get the same answer a majority of times (this is an idea stolen from the CoT-SC paper, chain of through self-consistency).
    • viraptor 2y ago
      It would be awesome if the author released a framework to play with this. I'd like to test things out, but I don't want to spend time redoing all his work from scratch.
      • fragmede 2y ago
        Just have ChatGPT write the framework
        • viraptor 2y ago
          If it takes so little time and is so trivial, you're welcome to send me a link to your generated solution.
• seizethecheese 2y ago
  All the hand wringing about openAI cheating suggests a question: why so much mistrust? My guess would be that the persona of the openAI team on platforms like Twitter is very cliquey. This, I think, naturally leads to mistrust. A clique feels more likely to cheat than some other sort of group.
  • simonw 2y ago
    I wrote about this last year. The levels of trust people have in companies working in AI is notably low: https://simonwillison.net/2023/Dec/14/ai-trust-crisis/ https://simonwillison.net/2023/Dec/14/ai-trust-crisis/
  • nuancebydefault 2y ago
    My take on this is that people tend to be afraid of what they can't understand or explain. To do away with that feeling, they just say 'it can't reason'. While nobody on earth can put a finger on what reasoning is, other than that it is a human trait.
• gallerdude 2y ago
  Very interesting - have you tried using `o1` yet? I made a program which makes LLM's complete WORDLE puzzles, and the difference between `4o` and `o1` is absolutely astonishing.
  • gallerdude 2y ago
    4o-mini: 16% 4o: 50% o1-mini: 97% o1: 100% * disclaimer - only n=7 on o1. Others are like 100-300 each
  • simonw 2y ago
    OK, that was fun. I just tried o1-preview on today's Wordle and it got it on the third guess: https://chatgpt.com/share/673f9169-3654-8006-8c0b-07c53a2c5881 https://chatgpt.com/share/673f9169-3654-8006-8c0b-07c53a2c58...
    • gallerdude 2y ago
      With some transcribing (using another LLM instance) I’ve even gotten it to solve NYT mini crosswords.
• ChrisArchitect 2y ago
  Related from last week: Something weird is happening with LLMs and Chess https://news.ycombinator.com/item?id=42138276 https://news.ycombinator.com/item?id=42138276
• kibwen 2y ago
  > I was astonished that half the internet is convinced that OpenAI is cheating. If you have a problem and all of your potential solutions are unlikely, then it's fine to assume the least unlikely solution while acknowledging that it's statistically probable that you're also wrong. IOW if you have ten potential solutions to a problem and you estimate that the most likely solution has an 11% chance of being true, it's fine to assume that solution despite the fact that, by your own estimate, you have an 89% chance of being wrong. The "OpenAI is secretly calling out to a chess engine" hypothesis always seemed unlikely to me (you'd think it would play much better, if so), but it seemed the easiest solution (Occam's razor) and I wouldn't have been surprised to learn it was true (it's not like OpenAI has a reputation of being trustworthy).
  • bongodongobob 2y ago
    That's not really how Occam's razor works. The entire company colluding and lying to the public isn't "easy". Easy is more along the lines of "for some reason it is good at chess but we're not sure why".
    • simonw 2y ago
      One of the reasons I thought that was unlikely was personal pride. OpenAI researchers are proud of the work that they do. Cheating by calling out to a chess engine is something they would be ashamed of.
      • kibwen 2y ago
        > OpenAI researchers are proud of the work that they do. Well, the failed revolution from last year combined with the non-profit bait-and-switch pretty much conclusively proved that OpenAI researchers are in it for the money first and foremost, and pride has a dollar value.
        • fkyoureadthedoc 2y ago
          How much say do individual researchers even have in this move? And how does that prove anything about their motivations "first and foremost"? They could be in it because they like the work itself, and secondary concerns like open or not don't matter to them. There's basically infinite interpretations of their motivations.
• tmalsburg2 2y ago
  Why not use temperature 0 for sampling? If the top-ranked move is not legal, it can’t play chess.
  • thornewolf 2y ago
    sometimes skilled chess players make illegal moves
    • atiedebee 2y ago
      Extremely rare. The only time this happened that I'm aware of was quite recent but the players only had a second or 2 remaining on the clock, so time pressure is definitely the reason there
      • GaggiX 2y ago
        It often happens when the players play blondfold chess, as in this case.
        • a2128 2y ago
          Is this really equivalent to blindfold chess? The LLM has access to the full move history, unlike blindfold chess where memorization is necessary
          • GaggiX 2y ago
            Even if memorization is not required, you still need to map every move on the board one by one under time pressure, an advance amateur would still make illegal moves.
• furyofantares 2y ago
  LLMs are fundamentally text-completion. The Chat-based tuning that goes on top of it is impressive but they are fundamentally text-completion, that's where most of the training energy goes. I keep this in mind with a lot of my prompting and get good results. Regurgitating and Examples are both ways to lean into that and try to recover whatever has been lost by Chat-based tuning.
  • zi_ 2y ago
    what else do you think about when prompting, which you've found to be useful?
• deleted 2y ago
  [deleted]
• blixt 2y ago
  Really interesting findings around fine-tuning. Goes to show it doesn't really affect the deeper "functionality" of the LLM (if you think of the LLM running a set of small functions on very high-dimensional numbers to produce a token). Using regurgitation to get around the assistant/user token separation is another fun tool for the toolbox, relevant for whenever you want a model that doesn't support continuation actually perform continuation (at the cost of a lot of latency). I wonder if any type of reflection or chains of thought would help it play better. I wouldn't be surprised if getting the LLM to write an analysis of the game in English is more likely to move it out of distribution than to make it pick better chess moves.
• MisterTea 2y ago
  This happened to a friend who was trying to sim basketball games. It kept forgetting who had the ball or outright made illegal or confusing moves. After a few days of wrestling with the AI he gave up. GPT is amazing at following a linear conversation but had no cognitive ability to keep track of a dynamic scenario.
• xg15 2y ago
  > In many ways, this feels less like engineering and more like a search for spells. This is still my impression of LLMs in general. It's amazing that they work, but for the next tech disruption, I'd appreciate something that doesn't make you feel like being in a bad sci-fi movie all the time.
• jey 2y ago
  Could be interesting to create a tokenizer that’s optimized for representing chess moves and then training a LLM (from scratch?) on stockfish games. (Using a custom tokenizer should improve the quality for a given size of the LLM model. So it doesn’t have to waste a lot of layers on encode and decode, and the “natural” latent representation is more straightforward)
• sourcepluck 2y ago
  I don't like being directly critical, people learning in public can be good and instructive. But I regret the time I've put into both this article and the last one and perhaps someone else can be saved the same time. This is someone with limited knowledge of chess, statistics and LLMs doing a series of public articles as they learn a little tiny bit about chess, statistics and LLMs. And it garners upvotes and attention off the coat-tails of AI excitement. Which is fair enough, it's the (semi-)public internet, but it sort of masquerades as being half-serious "research", and it kind of held things together for the first article, but this one really is thrown together to keep the buzz going of the last one. The TL;DR :: one of the AIs being just-above-terrible, compared to all the others being completely terrible, a fact already of dubious interest, is down to - we don't know. Maybe a difference in training sets. Tons of speculation. A few graphs.
• phkahler 2y ago
  You can easily construct a game board from a sequence of moves by maintaining the game state somewhere. But you can also know where a piece is bases on only its last move. I'm curious what happens if you don't feed it a position, but feed it a sequence of moves including illegal ones but end up at a given valid position. The author mention that LLMs will play differently when the same position is arrived at via different sequences. I'm suggesting to really play with that by putting illegal moves in the sequence. I doubt it's doing much more than a static analysis of the a board position, or even moving based mostly on just a few recent moves by key pieces.
• drivingmenuts 2y ago
  Why would a chess-playing AI be tuned to do anything except play chess? Just seems like a waste. A bunch of small, specialized AI's seems like a better idea than spending time trying to build a new one. Maybe less morally challenging, as well. You wouldn't be trying to install "sentience".
  • int_19h 2y ago
    It's the other way around - you might want a general-purpose model to learn to play chess because e.g. it improves its ability to reason logically in other cases (which has been claimed for humans pretty much ever since chess was invented). Considering that training models on code seems to improve their abilities on non-coding tasks in actual testing, this isn't even all that far-fetched. Perhaps that is why GPT-3.5 was specifically trained on chess in the first place.
• PaulHoule 2y ago
  People have to quit this kind of stumbling in the dark with commercial LLMs. To get to the bottom of this it would be interesting to train LLMs on nothing but chess games (can synthesize them endlessly by having Stockfish play against itself) with maybe a side helping of chess commentary and examples of chess dialogs “how many pawns are on the board?”, “where are my rooks?”, “draw the board”, competence at which would demonstrate that it has a representation of the board. I don’t believe in “emergent phenomena” or that the general linguistic competence or ability to feign competence is necessary for chess playing (being smart at chess doesn’t mean you are smart at other things and vice versa). With experiments like this you might prove me wrong though. This paper came out about a week ago https://arxiv.org/pdf/2411.06655 https://arxiv.org/pdf/2411.06655 seems to get good results with a fine-tuned Llama. I also like this one as it is about competence in chess commentary https://arxiv.org/abs/2410.20811 https://arxiv.org/abs/2410.20811
  • toxik 2y ago
    Predicting next moves of some expert chess policy is just imitation learning, a well-studied proposal. You can add return-to-go to let the network try to learn what kinds of moves are made in good vs bad games, which would be an offline RL regime (eg, Decision Transformers). I suspect chess skill is completely useless for LLMs in general and not an emergent phenomenon, just consuming gradient bandwidth and parameter space to do this neat trick. This is clear to me because the LLMs that aren't trained specifically on chess do not do chess well.
    • PaulHoule 2y ago
      In either language or chess I'm still a bit baffled how a representation over continuous variables (differentiable no less) works for something that is discrete such as words, letters, chess moves, etc. Add the word "not" a sentence and it is not a perturbation of the meaning but a reversal (or is it?) A difference between communication and chess is that your partner in conversation is your ally in meaning making and will help fix your mistakes which is how they get away with bullshitting. ("Personality" makes a big difference, by the time you are telling your programming assistant "Dude, there's a red squiggle on line 92" you are under its spell) Chess on the other hand is adversarial and your mistakes are just mistakes that your opponent will take advantage of. If you make a move and your hunch that your pieces are not in danger is just slightly wrong (one piece in danger) that's almost as bad as having all your non-King pieces in danger (they can only take one next turn.)
• joshka 2y ago
  Why are you telling it not to explain? Allowing the LLM space to "think" may be helpful, and would be definitely worth explorying? Why are you manually guessing ways to improve this? Why not let the LLMs do this for themselves and find iteratively better prompts?
• bambax 2y ago
  Very good follow-up to the original article. Thank you!
• deleted 2y ago
  [deleted]
• kqr 2y ago
  I get that it would make evals even more expensive, but I would also try chain-of-thought! Have it explain its goals and reasoning for the next move before making it. It might be an awful idea for something like chess, but it seems to help elsewhere.
• Palmik 2y ago
  It might be worth trying the experiment where the prompt is formatted such that each chess turn corresponds to one chat message.
• Jean-Papoulos 2y ago
  >According to that figure, fine-tuning helps. And examples help. But it’s examples that make fine-tuning redundant, not the other way around. This is extremely interesting. In this specific case at least, simply giving examples is equivalent to fine-tuning. This is a great discovery for me, I'll try using examples more often.
  • jdthedisciple 2y ago
    To me this is very intuitively true. I can't explain why.I always had the intuition that fine-tuning was overrated. One reason perhaps is that examples are "right there" and thus implicitly weighted much more in relation to the fine-tuned neurons.
  • s5ma6n 2y ago
    Agreed on providing examples is definitely a useful insight vs fine-tuning. While it is not very important for this toy case, it's good to keep in mind that each provided example in the input will increase the prediction time and cost compared to fine-tuning.
• marcus_holmes 2y ago
  I notice there's no prompt saying "you should try to win the game" yet the results are measured by how much the LLM wins. Is this implicit in the "you are a grandmaster chess player" prompt? Is there some part of the LLM training that does "if this is a game, then I will always try to win"? Could the author improve the LLM's odds of winning just by telling it to try and win?
  • Nashooo 2y ago
    IMO this is clearly implicit in the "you are a grandmaster chess player" prompt. As that should make generating best possible move tokens more likely.
    • Ferret7446 2y ago
      Is it? What if the AI is better than a grandmaster chess player and is generating the most likely next move that a grandmaster chess player might make and not the most likely move to win, which may be different?
      • lukan 2y ago
        Depends on the training data I think. If the data divides in games by top chess engines - and human players, then yes, it might make a difference to tell it, to play like a grandmaster of chess vs. to play like the top chess engine.
    • cma 2y ago
      Grandmasters usually play grandmasters of similar ELO, so it might think it doesn't always win. Even if it should recognize the player isn't a grandmaster, it still may be better to include that, though who knows without testing.
  • tinco 2y ago
    I think you're putting too much weight on its intentions, it doesn't have intentions it is a mathematical model that is trained to give the most likely outcome. In almost all examples and explanations it has seen from chess games, each player would be trying to win, so it is simply the most logical thing for it to make a winning move. So I wouldn't expect explicitly prompting it to win to improve its performance by much if at all. The reverse would be interesting though, if you would prompt it to make losing/bad moves, would it be effective in doing so, and would the moves still be mostly legal? That might reveal a bit more about how much relies on concepts it's seen before.
• byyoung3 2y ago
  sometimes new training techniques will lead to regressions in certain tasks. My guess is this is exactly what has happened.
• boesboes 2y ago
  It would be interesting to see if it can also play chess with altered rules, or actually just a novel 'game' that relies on logic & reasoning. Still not sure if that would 'prove' LLMs do reasoning, but I'd be pretty close to convinced.
  • blueboo 2y ago
    Fun idea. Let’s change how the knight behaves. Or try it on Really Bad Chess (puzzles with impossible layouts) or 6x6 chess or 8x9 chess. I wonder if there are variants that have good baselines. It might be tough to evaluate vis a vis human performance on novel games..
  • Miraltar 2y ago
    If they were trained on multiple chess variants that might work but as is it's impossible I think. Their internal model to play chess is probably very specific
• aaron695 2y ago
  [dead]
• leumassuehtam 2y ago
  I'm convinced that "completion" models are much more useful (and smart) than "chat" models, being able to provide more nuanced and original outputs. When gpt4 come out, text-davinci-003 would still provide better completions with the correct prompt. Of course this model was later replaced by gpt-3.5-turbo-instruct which is explored in this post. I believe the reason why such models were later deprecated was "alignment".
  • a2128 2y ago
    I don't believe alignment/safety is the only reason. You also burn through significantly more output tokens in a back-and-forth editing session because by default it keeps repeating the entire code or document just to make one small change, and it also adds useless fluff around the text ("You are absolutely correct, and I apologize for the confusion...")
• qnleigh 2y ago
  Two other theories that could explain why OpenAI's models do so well: 1. They generate chess games from chess engine self play and add that to the training data (similar to the already-stated theory about their training data). 2. They have added chess reinforcement learning to the training at some stage, and actually got it to work (but not very well).
• sourcepluck 2y ago
  > Since gpt-3.5-turbo-instruct has been measured at around 1800 Elo Where's the source for this? What's the reasoning? I don't see it. I have just relooked, and stil l can't see it. Is it 1800 lichess "Elo", or 1800 FIDE, that's being claimed? And 1800 at what time control? Different time controls have different ratings, as one would imagine/hope the author knows. I'm guessing it's not 1800 FIDE, as the quality of the games seems far too bad for that. So any clarity here would be appreciated.
  • famouswaffles 2y ago
    https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval
    • sourcepluck 2y ago
      Thank you. I had seen that, and had browsed through it, and thought: I don't get it, the reason for this 1800 must be elsewhere. What am I missing? Where does it show there how the claim of "1800 ELO" is arrived at? I can see various things that might be relevant, for example, the graph where it (GPT-3.5-turbo-instruct) is shown as going from mostly winning to mostly losing when it gets to Stockfish level 3. It's hard (/impossible) to estimate the lichess or FIDE ELO of the different Stockfish levels, but Lichess' Stockfish on level 3 is miles below 1800 FIDE, and it seems to me very likely to be below lichess 1800. I invite any FIDE 1800s and (especially) any Lichess 1800s to play Stockfish level 3 and report back. Years ago when I played a lot on Lichess I was low 2000s in rapid, and I win comfortably up till Stockfish level 6, where I can win, but also do lose sometimes. Basically I really have to start paying attention at level 6. Level 3 seems like it must be below lichess 1800, but it's just my anecdotal feeling of the strengths. Seeing as how the article is chocabloc full of unfounded speculation and bias though, maybe we can indulge ourselves too. So: someone please explain the 1800 thing to me? And any lichess 1800s like to play guinea pig, and play a series of games against stockfish 3, and report back to us?
      • famouswaffles 2y ago
        In Google's paper, then titled "Grandmaster level chess without search", they evaluate turbo-instruct to have a lichess Elo of 1755 (vs bots) https://arxiv.org/abs/2402.04494 https://arxiv.org/abs/2402.04494 Admittedly, this isn't really "the source" though. The first people to break the news on turbo-instruct's chess ability all pegged it around 1800. https://x.com/GrantSlatton/status/1703913578036904431 https://x.com/GrantSlatton/status/1703913578036904431
• sourcepluck 2y ago
  > For one, gpt-3.5-turbo-instruct rarely suggests illegal moves, even in the late game. It's claimed that this model "understands" chess, and can "reason", and do "actual logic" (here in the comments). I invite anyone making that claim to find me an "advanced amateur" (as the article says of the LLM's level) chess player who ever makes an illegal move. Anyone familiar with chess can confirm that it doesn't really happen. Is there a link to the games where the illegal moves are made?
  • zarzavat 2y ago
    An LLM is essentially playing blindfold chess if it just gets the moves and not the position. You have to be fairly good to never make illegal moves in blindfold.
    • fmbb 2y ago
      Does it not always have a list of all the moves in the game always at hand in the prompt? You have to give this human the same log of the game to refer to.
      • xg15 2y ago
        I think even then it would still be blindfold chess, because humans do a lot of "pattern matching" on the actual board state in front of them. If you only have the moves, you have to reconstruct this board state in your head.
    • pera 2y ago
      A chat conversation where every single move is written down and accessible at any time is not the same as blindfold chess.
      • zbyforgotp 2y ago
        You can make it available to the player and I suspect it wouldn’t change the outcomes.
      • 2y ago
• torginus 2y ago
  Sorry - I have a somewhat question - is it possible to train models as instruct models straight away? Previously LLMs were trained on raw text data, but now we can generate instruct data directly either from 'teaching LLMs' or ask existing LLMs to conver raw data into instruct format. Or alternatively - if chat tuning diminishes some of the models' capability, would it make sense to have a smaller chat model prompt a large base model, and convert back the outputs?
  • DHRicoF 2y ago
    I don't think there is enough (non syntetic) data available to get near what we are used to. The big breakthrough of GPT was exactly that. You can train a model with (for what that time was) stupidly high amount of data and make it okis to a lot of task you haven't trained explicitly.
    • torginus 2y ago
      You can make GPT rewrite all existing textual info into chatbot format, so there's no loss there. With newer techniques, such as chain of thought and self-checking, you can also generate a ton of high-quality training data, that won't degrade the output of the LLM. Though the degree to which you can do that is not clear to me. Imo it makes sense to train an LLM as a chatbot from the start.
• GaggiX 2y ago
  You should not finetune the models on the strongest setting of Stockfish as the move will not be understandable unless you really dig deep into the position and the model would not be able to find a pattern to make sense of it, instead I suggest training on human games of a certain ELO (less than grandmaster).
• codeflo 2y ago
  > everyone is wrong! Well, not everyone. I wasn't the only one to mention this, so I'm surprised it didn't show up in the list of theories, but here's e.g. me, seven days ago (source https://news.ycombinator.com/item?id=42145710 https://news.ycombinator.com/item?id=42145710): > At this point, we have to assume anything that becomes a published benchmark is specifically targeted during training. This is not the same thing as cheating/replacing the LLM output, the theory that's mentioned and debunked in the article. And now the follow-up adds weight to this guess: > Here’s my best guess for what is happening: ... OpenAI trains its base models on datasets with more/better chess games than those used by open models. ... Meanwhile, in section A.2 of this paper (h/t Gwern) some OpenAI authors mention that GPT-4 was trained on chess games in PGN notation, filtered to only include players with Elo at least 1800. To me, it makes complete sense that OpenAI would "spike" their training data with data for tasks that people might actually try. There's nothing unethical about this. No dataset is ever truly "neutral", you make choices either way, so why not go out of your way to train the model on potentially useful answers?
  • stingraycharles 2y ago
    Yup, I remember reading your comment and that making the most sense to me. OpenAI just shifted their training targets, initially they thought Chess was cool, maybe tomorrow they think Go is cool, or maybe the ability to write poetry. Who knows. But it seems like the simplest explanation and makes the most sense.
    • qup 2y ago
      At current sizes, these things are like humans. They gotta specialize. Maybe that'll be enough moat to save us from AGI.
  • demaga 2y ago
    Yes, and I would like this approach to also be used in other, more practical areas. I mean, more "expert" content than "amateur" content in training data, regardless of area of expertise.
  • dr_dshiv 2y ago
• deadbabe 2y ago
  If you randomly position pieces on the board and then ask the LLM to play chess, where each piece still moves according to its normal rules, does it know how to play still?
  • Hilift 2y ago
    If the goal is to create a model that simulates normal human intelligence, yes. Some try to measure accuracy or performance based on expertise though.
    • deadbabe 2y ago
      It does not.
• keskival 2y ago
  "I’m not sure, because OpenAI doesn’t deign to share gpt-4-base, nor to allow queries of gpt-4o in completion mode." I would guess GPT-4o isn't first pre-trained and then instruct-tuned, but trained directly with refined instruction-following material. This material probably contains way fewer chess games.
  • toxik 2y ago
    Why do you think that? InstructGPT was predominantly trained as a next-token predictor on whatever soup of data OpenAI curated at the time. The alignment signal (both RL part and the supervised prompt/answer pairs) are a tiny bit of the gradient.
• wavemode 2y ago
  I have the exact same problem with this article that I had with the previous one - the author fails to provide any data on the frequency of illegal moves. Thus it's impossible to draw any meaningful conclusions. It would be similar to if I claimed that an LLM is an expert doctor, but in my data I've filtered out all of the times it gave incorrect medical advice.
  • falcor84 2y ago
    I world argue that it's more akin to filtering out the chit-chat with the patient, where the doctor explained things in an imprecise manner, keeping only the formal and valid medical notation
    • caddemon 2y ago
      There is no legitimate reason to make an illegal move in chess though? There are reasons why a good doctor might intentionally explain things imprecisely to a patient.
      • hnthrowaway6543 2y ago
        > There is no legitimate reason to make an illegal move in chess though? If you make an illegal move and the opponent doesn't notice it, you gain a significant advantage. LLMs just have David Sirlin's "Playing to Win" as part of their training data.
        • fluoridation 2y ago
          You raise an interesting point. If the filtered out illegal moves were disadvantageous, it could be that if the model had been allowed to make any moves it wanted it would have played to a much worse level than it did.
    • ses1984 2y ago
      It’s like the doctor saying, “you have cancer? Oh you don’t? Just kidding. Parkinson’s. Oh it’s not that either? How about common cold?”
• tech_ken 2y ago
  > It’s ridiculously hard to find the optimal combination of prompts and examples and fine-tuning, etc. It’s a very large space, there are no easy abstractions to allow you to search through the space, LLMs are unpredictable and fragile, and these experiments are slow and expensive. Regardless of the actual experiment outcome, I think this is a super valuable insight. "Should we provide legal moves?" section is an excellent case study of this- extremely prudent idea actually degrades model performance, and quite badly. It's like that crocodile game where you're pushing teeth until it clamps onto your hand.
• subarctic 2y ago
  The author either didn't read the hacker news comments last time, or he missed the top theory that said they probably used chess as a benchmark when they developed the model that is good at chess for whatever business reasons they had at the time.
  • wavemode 2y ago
    This is plausible. One of the top chess engines in the world (Leela) is just a neural network trained on billions of chess games. So it makes sense that an LLM would also be able to acquire some skill by simply having a large volume of chess games in its training data. OpenAI probably just eventually decided it wasn't useful to keep pursuing chess skill.
  • devindotcom 2y ago
    fwiw this is exactly what i thought - oai pursued it as a skillset (likely using a large chess dataset) for their own reasons and then abandoned it as not particularly beneficial outside chess. It's still interesting to try to replicate how you would make a generalist LLM good at chess, so i appreciated the post, but I don't think there's a huge mystery!
  • brcmthrowaway 2y ago
    Oh really! What happened to the theory that training on code magically caused some high level reasoning ability?
• koolala 2y ago
  Next test a image & text model! Chess is way easier when you can see the board.
• amelius 2y ago
  I wonder what would happen if they changed the prompt such that the llm is asked to explain their strategy first. Or to explain their opponent's strategy.
• code51 2y ago
  Initially LLM researchers were saying training on code samples made the "reasoning" better. Now, if "language to world model" thesis is working, shouldn't chess actually be the smallest case for it? I can't understand why no research group is going hard at this.
  • throwaway314155 2y ago
    I don't think training on code and training on chess are even remotely comparable in terms of available data and linguistic competency required. Coding (in the general case, which is what these models try to approach) is clearly the harder task and contains _massive_ amounts of diverse data. Having said all of that, it wouldn't surprise me if the "language to world model" thesis you reference is indeed wrong. But I don't think a model that plays chess well disproves it, particularly since there are chess engines using old fashioned approaches that utterly destroy LLM's.
• bee_rider 2y ago
  Extremely tangential, but how to chess engines do when playing from illegal board states? Could the LLM have a chance of competing with a real chess engine from there? Understanding is a funny concept to try to apply to computer programs anyway. But playing from an illegal state seems (to me at least) to indicate something interesting about the ability to comprehend the general idea of chess.
  • layman51 2y ago
    I just checked the Lichess board editor tool and it won’t let you continue analyzing with an engine if you have an illegal board setup (like the two kings adjacent to each other). I haven’t tried this yet, but I think you can set up and analyze board positions that are legal but that could never be reached in a real game (e.g, having nine pawns of one color).
    • bee_rider 2y ago
      Oh, I was thinking board positions that could never be reached in a real game. Actually, I have no idea what the technical definition of a legal board state is, haha.
      • layman51 2y ago
        Well, there are chess problems that are specifically composed so that a solver can work out the right moves to achieve checkmate. For a lot of these composed chess positions, it’s virtually impossible for to get them in a real game but that’s not the point. Engines can analyze these but they can get tripped up depending on how the problem was designed.[1] The legality of the board state seems like a different concept. An engine can’t be used to analyze a position of where the Black king is in check and it is White’s turn to move. It doesn’t matter if you got to that board state by composing it or by making a sequence of previously legal moves. [1]: https://chess.stackexchange.com/questions/23837/solvable-studies-that-computers-fail-to-crack https://chess.stackexchange.com/questions/23837/solvable-stu...
• derefr 2y ago
  > Many, many people suggested that there must be some special case in gpt-3.5-turbo-instruct that recognizes chess notation and calls out to an external chess engine. Not that I think there's anything inherently unreasonable about an LLM understanding chess, but I think the author missed a variant hypothesis here: What if that specific model, when it recognizes chess notation, is trained to silently "tag out" for another, more specialized LLM, that is specifically trained on a majority-chess dataset? (Or — perhaps even more likely — the model is trained to recognize the need to activate a chess-playing LoRA adapter?) It would still be an LLM, so things like "changing how you prompt it changes how it plays" would still make sense. Yet it would be one that has spent a lot more time modelling chess than other things, and never ran into anything that distracted it enough to catastrophically forget how chess works (i.e. to reallocate some of the latent-space vocabulary on certain layers from modelling chess, to things that matter more to the training function.) And I could certainly see "playing chess" as a good proving ground for testing the ability of OpenAI's backend to recognize the need to "loop in" a LoRA in the inference of a response. It's something LLM base models suck at; but it's also something you intuitively could train an LLM to do (to at least a proficient-ish level, as seen here) if you had a model focus on just learning that. Thus, "ability of our [framework-mediated] model to play chess" is easy to keep an eye on, long-term, as a proxy metric for "how well our LoRA-activation system is working", without needing to worry that your next generation of base models might suddenly invalidate the metric by getting good at playing chess without any "help." (At least not any time soon.)
  • throwaway314155 2y ago
    > but I think the author missed a variant hypothesis here: > What if that specific model, when it recognizes chess notation, is trained to silently "tag out" for another, more specialized LLM, that is specifically trained on a majority-chess dataset? (Or — perhaps even more likely — the model is trained to recognize the need to activate a chess-playing LoRA adapter?) Pretty sure your variant hypothesis is sufficiently covered by the author's writing. So strange that people are so attached to conspiracy theories in this instance. Why would OpenAI or anyone go through all the trouble? The proposals outlined in the article make far more sense and track well with established research (namely that applying RLHF to a "text-only" model tends to wreak havoc on said model).
• bob1029 2y ago
  I find it amusing that we would frame an ensemble of models as "cheating". Routing to a collection of specialized models via classification layers seems like the most obvious path for adding practical value to these solutions. Why conflate the parameters of chess with checkers and go if you already have high quality models for each? I thought tool use and RAG were fair game.
• copperroof 2y ago
  I just want a hacker news no-LLM filter. The site has been almost unusable for a year now.
• XenophileJKO 2y ago
  So this article is what happens when people who don't really understand the models "test" things. There are several fatal flaws. The first problem is that he isn't clearly and concisely displaying the current board state. He is expecting the model to attend a move sequence to figure out the board state. Secondly, he isn't allowing the model to think elastically using COT or other strategies. Honestly, I am shocked it is working at all. He has basically formulated the problem in the worst possible way.
  • yeevs 2y ago
    I'm not sure COT would help in this situation. I am an amateur at chess but in my experience a large part of playing is intuition and making and I'm not confident the model could even accurately summarise its thinking. There are tasks in which models perform worse on when explaining reasoning. However, this is completely vibes based.
    • XenophileJKO 2y ago
      Given my experience with the models, giving it the ability to think would allow it to attend to different ramifications of the current board layout. I would expect a non trivial performance gain.
• cma 2y ago
  One thing missing from the graphs is whether 3.5-turbo-instruct also gets better with the techniques? Is finetuning available for it?
• __MatrixMan__ 2y ago
  It would be fun to play against an LLM without having to think about the prompting, if only as a novel way to get a "feel" for how they "think".
• timzaman 2y ago
  "all LLMs" - OP only tested OpenAI LLMs. Try Gemini.
• elif 2y ago
  The problem I see with this prompting is he is going to great lengths instructing the LLM that he is a grandmaster and makes great moves etc. but nowhere in the prompt does he instruct the LLM to attempt to win the game of chess. It may just be coming up with "good moves" that a "grandmaster" would make, but after all, grandmasters still lose 49% of their games, all the while making "good moves" I would suppose that the LLM is actually wholly uninterested in victory given this prompting.
• fijiaarone 2y ago
  It's very clear that GPT-3.5-turbo-whatever is cheating and that LLMs cannot, in fact play chess. It was trained in sequences of chess moves and has explicit coding to recognize chess moves and respond accordingly. If you only define "cheating" as calling a chess engine like stockfish, then your definition of cheating is too narrow. It's exactly like the strawberry problem. No LLM can count the letters in a word. But when shown that, they explicitly were taught to recognize the prompt and count the letters in the word. They didn't create a letter counting algorithm, but they did build a table of words and letter counts. And every "new" LLM explicitly looks for a phrase that looks like "how many Rs are in stawberry" and then the LLM looks in the "letters in words" neural network instead of the the "what is the next likely word in this sentence net". All "new" LLMs (in the next few weeks) will suddenly become decent at chess because they will have weighted preference to look at the "chess moves" neural net instead of the "random numbers and letters sequence" neural net when they detect a sentence that looks like "d4, d5; nd3, ?" etc.
• Animats 2y ago
  The main insight is that the LLM has to be trained on good chess games. If the training set is of random games, that's not helpful. I'm still amazed that this works at all, since it lacks actual board state.
• throw310822 2y ago
  > For one, gpt-3.5-turbo-instruct rarely suggests illegal moves, even in the late game. This requires “understanding” chess. If this doesn’t convince youI encourage you to write a program that can take strings like 1. e4 d5 2. exd5 Qxd5 3. Nc3 and then say if the last move was legal. This alone should put to rest all the arguments that LLMs lack a world model, that they're just manipulating words, that they're just spitting probabilistic answers, and so on.
• herbst 2y ago
  Is it just me or is the answer quality rapidly decreasing for the public models anyway (o-mini and whatever it degrades to) I am shopping things for a bigger DIY project for a few months now and recently it started to hallucinate products with the specifications I need. In fact it returns mainly broken code, hallucinates functions that never existed (zero Google results) and so on. Not sure if I am using it more, or it just got much more useless.