4 ms·

Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to

Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.

willmarch 10d ago
If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn). Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
- rsfern 10d ago
  the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
  - willmarch 10d ago
    That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
    - rsfern 10d ago
      Maybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)
      - willmarch 8d ago
        Your child never messed up en passant (or had trouble understanding it in a real game), castled through or into a check, didn't see a discovered check after moving a piece, never got confused by how stalemate works?
- what 10d ago
  > considering LLMs currently play better than a brand new human player would They’ve ingested all the literature on playing chess, a brand new human player has not.
  - willmarch 10d ago
    Yes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either). We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
    - skydhash 10d ago
      > The only way you get better at chess is by playing a lot of games and learning from mistakes How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice. This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
      - willmarch 10d ago
        Reading can kickstart the process, but you can also make random moves guided by some sort of system (such as a computer GUI) or learn by watching other players play. The overall point is that you learn through observation and lots of trial and error (whether you are a human or a computer). And beginners in chess often make illegal moves even after learning the rules, it's fairly common. It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
        10d ago
- orwin 10d ago
  That's quite untrue. I taught my (adult) brother the moves, the only illegal move he ever tried against me (over his 6 first games) was a castle with a rook that already moved twice. Within a few hundred games (less than 500 for sure, he played 3 minutes blitz but always took at least 10 minutes analyzing his games) he was rated 1100 on lichess (which is like 1050 on chess.com and unranked in the real world).
  - willmarch 10d ago
    So your brother tried to make illegal moves while learning the game and it took your brother hundreds of games to get to be a decent player? I don't see how this contradicts anything I said...
    - orwin 10d ago
      The _only_ illegal move a human might make as a beginner is a failed en passant or a bad castle. And yes, a few hundred games is all it takes to be better than any publicly available LLM at the moment.
      - willmarch 8d ago
        ...and mistakes like not seeing discovered checks, castling into check, trying to castle out of check, castling through a check, missing checks when promoting a piece, misunderstanding how stalemate works, etc? If you compared a human after hundreds of games to a SOTA LLM that was also trained on the output of hundreds of chess games that it played, I suspect you would notice similar improvements.
        orwin 8d ago
        Honestly, if we're only talking about SOTA llms, in sandbox mode without harness? No shot. Without harness LLMs have no memory of previous moves. I can't make the last version of chatgpt remember more than 5 movements. I guarantee you, if you do not add flags in the harness with 'left rook moved' or 'right rook moved', it will try to illegally castle 100% of the time it's in the situation. Llm+harness, just make it call stockfish tbh.
hi_im_greg_h 10d ago
1. The LLMs have surely ingested hundreds if not thousands of books on chess. 2. The study (along with other posters here) show the models can’t even stick to following the rules of the game

willmarch 10d ago

If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn). Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).

rsfern 10d ago
the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
- willmarch 10d ago
  That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
  - rsfern 10d ago
    Maybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)
    - willmarch 8d ago
      Your child never messed up en passant (or had trouble understanding it in a real game), castled through or into a check, didn't see a discovered check after moving a piece, never got confused by how stalemate works?
what 10d ago
> considering LLMs currently play better than a brand new human player would They’ve ingested all the literature on playing chess, a brand new human player has not.
- willmarch 10d ago
  Yes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either). We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
  - skydhash 10d ago
    > The only way you get better at chess is by playing a lot of games and learning from mistakes How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice. This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
    - willmarch 10d ago
      Reading can kickstart the process, but you can also make random moves guided by some sort of system (such as a computer GUI) or learn by watching other players play. The overall point is that you learn through observation and lots of trial and error (whether you are a human or a computer). And beginners in chess often make illegal moves even after learning the rules, it's fairly common. It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
      - 10d ago
orwin 10d ago
That's quite untrue. I taught my (adult) brother the moves, the only illegal move he ever tried against me (over his 6 first games) was a castle with a rook that already moved twice. Within a few hundred games (less than 500 for sure, he played 3 minutes blitz but always took at least 10 minutes analyzing his games) he was rated 1100 on lichess (which is like 1050 on chess.com and unranked in the real world).
- willmarch 10d ago
  So your brother tried to make illegal moves while learning the game and it took your brother hundreds of games to get to be a decent player? I don't see how this contradicts anything I said...
  - orwin 10d ago
    The _only_ illegal move a human might make as a beginner is a failed en passant or a bad castle. And yes, a few hundred games is all it takes to be better than any publicly available LLM at the moment.
    - willmarch 8d ago
      ...and mistakes like not seeing discovered checks, castling into check, trying to castle out of check, castling through a check, missing checks when promoting a piece, misunderstanding how stalemate works, etc? If you compared a human after hundreds of games to a SOTA LLM that was also trained on the output of hundreds of chess games that it played, I suspect you would notice similar improvements.
      - orwin 8d ago
        Honestly, if we're only talking about SOTA llms, in sandbox mode without harness? No shot. Without harness LLMs have no memory of previous moves. I can't make the last version of chatgpt remember more than 5 movements. I guarantee you, if you do not add flags in the harness with 'left rook moved' or 'right rook moved', it will try to illegally castle 100% of the time it's in the situation. Llm+harness, just make it call stockfish tbh.

hi_im_greg_h 10d ago

1. The LLMs have surely ingested hundreds if not thousands of books on chess. 2. The study (along with other posters here) show the models can’t even stick to following the rules of the game