4 ms·

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

scarmig 4y ago
We would expect some vanishingly small fraction. Even at the grandmaster level, they happen (rarely): https://chess24.com/en/read/news/the-7-most-illegal-chess-moves-of-all-time https://chess24.com/en/read/news/the-7-most-illegal-chess-mo...
famouswaffles 4y ago
Not if training is unsupervised. If you've never been explicitly told the rules of game, you can never be 100% sure of all possible illegal moves. anyway the 3.5 series can't ply chess but gpt-4 certainly can.
- Jensson 4y ago
  > anyway the 3.5 series can't ply chess but gpt-4 certainly can. This article stated the opposite, gpt-4 couldn't play chess while gpt-3.5 could. So this is a case where the model got dumber.
  - famouswaffles 4y ago
    Not sure what's up but from here https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_to_improve_chess_skills_significantly/ https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_t... nd here https://twitter.com/zswitten/status/1631107663500304384 https://twitter.com/zswitten/status/1631107663500304384 Gpt-4 plys chess just fine.
- rightbyte 4y ago
  Surely the rules of chess has to be in the training data 100s of times? Maybe it is not enough.
bloppe 4y ago
This. The author is very generous with their interpretation: > I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” Kind of sounds like anthropomorphization, but more likely the author just papering over the glaring shortcomings to produce a compelling blog post. It also sounds like the illegal moves were rather frequent. The 61-legal-move game sounded like an impressive outlier.
- rightbyte 4y ago
  I think the authors rule is fair. If we interpret illegal moves as getting stuck in an online game, the resulting Elo rating is what it would get. But ye, he is anthropomorphizing alot ...
  - bloppe 4y ago
    There's no indication that GPT-3.5 was stuck when it tried to make illegal moves. GPT-4 clearly was making illegal moves when it was very much not stuck. It just doesn't know how to play, but the author decided to interpret it as frustration.
    - rightbyte 4y ago
      I meant stuck as "it can't make the move" and would timeout the game if playing online.
oezi 4y ago
I think there is very low percentage of players at elo 1400 who can provide a valid next move after seeing just the list of moves and not the current board state.
- dragontamer 4y ago
  I'm Elo 1400 and can beat literally everyone I know in the real world. I need to go online to find players at my skill level, or find tournament/competitive settings for a challenge. Yeah, I'm "class C", weak amateur chess player, but I think you're grossly underestimating the amount of study I put into this game. I'm not going to make an illegal move
  - oezi 4y ago
    I mean can you play just based on being provided the input of a series of moves without it being shown to you as a visual board? I guess most players would mess up 20/30 moves in.
    - dragontamer 4y ago
      https://chessily.com/blog/stockfish-vs-chatgpt/ https://chessily.com/blog/stockfish-vs-chatgpt/ > ChatGPT: Yes, that’s a good move for you. My next move is: Bc3, developing my pieces and attacking your pawn on c3. I am 1400 Elo and can tell you that from an near opening position, its impossible to move a Bishop to c3 for either Black or White in the first say, 10 moves, under traditional openings.
  - medvezhenok 4y ago
    You will under time pressure :) even Grandmasters have done that (I'm around 1900 elo for context) Also people forgetting they moved the king/rook and trying to castle.
    - dragontamer 4y ago
      Watch this ChatGPT game. https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... We're talking about pieces that don't exist, reappearing pieces, pieces moving completely wrong (Knight takes as if its a Pawn), etc. etc. --------- People are taking these example games and saying ChatGPT is 1400 strength. I don't think so. This isn't a case of "oops, I castled even though I moved my king 15 turns ago".

We would expect some vanishingly small fraction. Even at the grandmaster level, they happen (rarely): https://chess24.com/en/read/news/the-7-most-illegal-chess-moves-of-all-time https://chess24.com/en/read/news/the-7-most-illegal-chess-mo...

famouswaffles 4y ago

Not if training is unsupervised. If you've never been explicitly told the rules of game, you can never be 100% sure of all possible illegal moves. anyway the 3.5 series can't ply chess but gpt-4 certainly can.

Jensson 4y ago
> anyway the 3.5 series can't ply chess but gpt-4 certainly can. This article stated the opposite, gpt-4 couldn't play chess while gpt-3.5 could. So this is a case where the model got dumber.
- famouswaffles 4y ago
  Not sure what's up but from here https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_to_improve_chess_skills_significantly/ https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_t... nd here https://twitter.com/zswitten/status/1631107663500304384 https://twitter.com/zswitten/status/1631107663500304384 Gpt-4 plys chess just fine.
rightbyte 4y ago
Surely the rules of chess has to be in the training data 100s of times? Maybe it is not enough.

bloppe 4y ago

This. The author is very generous with their interpretation: > I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” Kind of sounds like anthropomorphization, but more likely the author just papering over the glaring shortcomings to produce a compelling blog post. It also sounds like the illegal moves were rather frequent. The 61-legal-move game sounded like an impressive outlier.

rightbyte 4y ago
I think the authors rule is fair. If we interpret illegal moves as getting stuck in an online game, the resulting Elo rating is what it would get. But ye, he is anthropomorphizing alot ...
- bloppe 4y ago
  There's no indication that GPT-3.5 was stuck when it tried to make illegal moves. GPT-4 clearly was making illegal moves when it was very much not stuck. It just doesn't know how to play, but the author decided to interpret it as frustration.
  - rightbyte 4y ago
    I meant stuck as "it can't make the move" and would timeout the game if playing online.

oezi 4y ago

I think there is very low percentage of players at elo 1400 who can provide a valid next move after seeing just the list of moves and not the current board state.

dragontamer 4y ago
I'm Elo 1400 and can beat literally everyone I know in the real world. I need to go online to find players at my skill level, or find tournament/competitive settings for a challenge. Yeah, I'm "class C", weak amateur chess player, but I think you're grossly underestimating the amount of study I put into this game. I'm not going to make an illegal move
- oezi 4y ago
  I mean can you play just based on being provided the input of a series of moves without it being shown to you as a visual board? I guess most players would mess up 20/30 moves in.
  - dragontamer 4y ago
    https://chessily.com/blog/stockfish-vs-chatgpt/ https://chessily.com/blog/stockfish-vs-chatgpt/ > ChatGPT: Yes, that’s a good move for you. My next move is: Bc3, developing my pieces and attacking your pawn on c3. I am 1400 Elo and can tell you that from an near opening position, its impossible to move a Bishop to c3 for either Black or White in the first say, 10 moves, under traditional openings.
- medvezhenok 4y ago
  You will under time pressure :) even Grandmasters have done that (I'm around 1900 elo for context) Also people forgetting they moved the king/rook and trying to castle.
  - dragontamer 4y ago
    Watch this ChatGPT game. https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_placed_stockfish_white_against_chatgpt_black/ https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... We're talking about pieces that don't exist, reappearing pieces, pieces moving completely wrong (Knight takes as if its a Pawn), etc. etc. --------- People are taking these example games and saying ChatGPT is 1400 strength. I don't think so. This isn't a case of "oops, I castled even though I moved my king 15 turns ago".