4 ms·

GPT-3.5-turbo-instruct can play chess around 1800 Elo

38 points by beefman 3y ago

djmips 3y ago
I'm sorry, Frank, I think you missed it. Queen to Bishop 3, Bishop takes Queen, Knight takes Bishop. Mate.
famouswaffles 3y ago
Yeah. But no Chain of Thought, no board descriptions etc. Just pure PGN completion.
- nextaccountic 3y ago
  Maybe asking the model to explain the moves would work?
  - famouswaffles 3y ago
    No i was saying those things seem to degrade performance. Just move completion is the best right now.
    - akomtu 3y ago
      That's odd. Ask it to generate 3 moves and then pick the best. Even better: ask it to generate 3 moves with counter-moves, and then pick the best move.
GaggiX 3y ago
What it surprised me the most is that according to the author the model do not make illegal moves meaning it has developed a pretty good model about the game (it's not given any representation about the board during the game). Even though GPT models should not really be that good to model the game of chess in general as any new tokens starts from zero and it needs to retrieve all information it needs, and the state of the board cannot be "saved" at the same layer because after the token has retrived it it would need to add the new semi-move on top and it's already one layer more deep, RNN seems they would be better at this as the cells take the h vector as input, add new information on it and pass it to the next tokens at the same layer, so it would be perfect for saving the state of the game; of course this would not allow parallel training like transformer models, this is the tradeoff.
solardev 3y ago
What is an Elo? Is 1800 average for a computer program or human? I read the wiki and apparently the famous grandmasters are like 2500+, but I dunno the distribution below that. https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system
- theonemind 3y ago
  Elo is just a rating point. There are a lot of details, but the main point is that it tries to predict probability of winning. If a player has a rating 200 elo higher than you, they should win about 75% of the time. When you win, your elo goes up, and the other player's elo goes down. Bigger initial differences result in bigger changes, blahblah, lot of details. it also helps rank the players because not every player just plays every other player, so you can't just have #1, #2, #3, etc. But you might consider the player with the highest elo the de-facto #1 player. But they haven't actually played every other player. So it also only works on a fixed pool or player base. So, you can't directly compare ratings from different organizations, etc. It doesn't actually measure a player's skill, it works closer to a stack ranking of a particular pool of players. A lot of people seem to think it measures "skill" directly in some way, but it doesn't do that in any direct way at all.
- brokensegue 3y ago
  1800 is bad for a computer but good for an amateur human
- famouswaffles 3y ago
  Good for a human (but not best of the best). Bad for a dedicated chess computer.
noman-land 3y ago
And what about off book?
famouswaffles 3y ago
There's a community project at the Eleuther discord training LLMs for chess on the pythia models (and maybe Tiny Stories). So far: Pythia 70M, ELO 1050 Pythia 160M, ELO 1370
seanhunter 3y ago
For a fun exercise, I tried Turing's chess problem[1] from "Computing Machinery and Intelligence"[2] on GPT-4. It recognized that it was a chess problem but actually failed to solve it in a really interesting way. The reason (I think) it failed to find the mate is that Turing gives the problem in a form of relative notation that noone would use any more, with the coordinates of the pieces given relative to the player. Nowdays everyone uses a form of notation which describes the squares the same for the player with black as white. Even if they didn't use algebraic notation (which everyone uses now), anyone giving the problem today would call the position of the black King K3 and the rook R8. When I prompt GPT4 using modern notation it realises it got the pieces positions wrong but still makes the same mistake of missing the very obvious mate, having got itself stuck. [3] I also tried putting the Black king in a different position where Rh1 would not be checkmate and was able to convince gpt4 that it was mate. All of this makes me certain it doesn't actually have a model of the game - it really just simulates a model of the game by text reconstruction, which is exactly what you would think. [1] "I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move, what do you play?" [2] https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf [3] Transcript here https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f531142ed90 https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f5311...
thethirdone 3y ago
It is important to note that the one game provided as evidence of 1800ish rating is an opening blunder by stockfish that has occurred in at least 8000 human games. I would expect GPT-3 to be trained on several explanations of how to punish this specific mistake. In general, I would expect GPT's openings to be higher quality than the rest of its games. The fact that it didn't play any illegal moves and was able to win once it had a +5 advantage is still enough to say it is at least 1400ish in the rest of the game. Some of the other games [0] that the OP imported would not strike me as 1800 level play. In particular, the endgame play is atrociously below that level. None of them are labeled as stockfish vs GPT though. [0]: https://lichess.org/@/grantslatton/imported https://lichess.org/@/grantslatton/imported
- gamegoblin 3y ago
  Lots of those are various experiments with different prompting strategies, models, etc, so not representative. It regularly beats Stockfish Level 4 even without opening blunders. source: I am OP

38 points by beefman 3y ago

djmips 3y ago

I'm sorry, Frank, I think you missed it. Queen to Bishop 3, Bishop takes Queen, Knight takes Bishop. Mate.

famouswaffles 3y ago

Yeah. But no Chain of Thought, no board descriptions etc. Just pure PGN completion.

nextaccountic 3y ago
Maybe asking the model to explain the moves would work?
- famouswaffles 3y ago
  No i was saying those things seem to degrade performance. Just move completion is the best right now.
  - akomtu 3y ago
    That's odd. Ask it to generate 3 moves and then pick the best. Even better: ask it to generate 3 moves with counter-moves, and then pick the best move.

GaggiX 3y ago

What it surprised me the most is that according to the author the model do not make illegal moves meaning it has developed a pretty good model about the game (it's not given any representation about the board during the game). Even though GPT models should not really be that good to model the game of chess in general as any new tokens starts from zero and it needs to retrieve all information it needs, and the state of the board cannot be "saved" at the same layer because after the token has retrived it it would need to add the new semi-move on top and it's already one layer more deep, RNN seems they would be better at this as the cells take the h vector as input, add new information on it and pass it to the next tokens at the same layer, so it would be perfect for saving the state of the game; of course this would not allow parallel training like transformer models, this is the tradeoff.

solardev 3y ago

What is an Elo? Is 1800 average for a computer program or human? I read the wiki and apparently the famous grandmasters are like 2500+, but I dunno the distribution below that. https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system

theonemind 3y ago
Elo is just a rating point. There are a lot of details, but the main point is that it tries to predict probability of winning. If a player has a rating 200 elo higher than you, they should win about 75% of the time. When you win, your elo goes up, and the other player's elo goes down. Bigger initial differences result in bigger changes, blahblah, lot of details. it also helps rank the players because not every player just plays every other player, so you can't just have #1, #2, #3, etc. But you might consider the player with the highest elo the de-facto #1 player. But they haven't actually played every other player. So it also only works on a fixed pool or player base. So, you can't directly compare ratings from different organizations, etc. It doesn't actually measure a player's skill, it works closer to a stack ranking of a particular pool of players. A lot of people seem to think it measures "skill" directly in some way, but it doesn't do that in any direct way at all.
brokensegue 3y ago
1800 is bad for a computer but good for an amateur human
famouswaffles 3y ago
Good for a human (but not best of the best). Bad for a dedicated chess computer.

noman-land 3y ago

And what about off book?

famouswaffles 3y ago

There's a community project at the Eleuther discord training LLMs for chess on the pythia models (and maybe Tiny Stories). So far: Pythia 70M, ELO 1050 Pythia 160M, ELO 1370

seanhunter 3y ago

For a fun exercise, I tried Turing's chess problem[1] from "Computing Machinery and Intelligence"[2] on GPT-4. It recognized that it was a chess problem but actually failed to solve it in a really interesting way. The reason (I think) it failed to find the mate is that Turing gives the problem in a form of relative notation that noone would use any more, with the coordinates of the pieces given relative to the player. Nowdays everyone uses a form of notation which describes the squares the same for the player with black as white. Even if they didn't use algebraic notation (which everyone uses now), anyone giving the problem today would call the position of the black King K3 and the rook R8. When I prompt GPT4 using modern notation it realises it got the pieces positions wrong but still makes the same mistake of missing the very obvious mate, having got itself stuck. [3] I also tried putting the Black king in a different position where Rh1 would not be checkmate and was able to convince gpt4 that it was mate. All of this makes me certain it doesn't actually have a model of the game - it really just simulates a model of the game by text reconstruction, which is exactly what you would think. [1] "I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move, what do you play?" [2] https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf [3] Transcript here https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f531142ed90 https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f5311...

thethirdone 3y ago

It is important to note that the one game provided as evidence of 1800ish rating is an opening blunder by stockfish that has occurred in at least 8000 human games. I would expect GPT-3 to be trained on several explanations of how to punish this specific mistake. In general, I would expect GPT's openings to be higher quality than the rest of its games. The fact that it didn't play any illegal moves and was able to win once it had a +5 advantage is still enough to say it is at least 1400ish in the rest of the game. Some of the other games [0] that the OP imported would not strike me as 1800 level play. In particular, the endgame play is atrociously below that level. None of them are labeled as stockfish vs GPT though. [0]: https://lichess.org/@/grantslatton/imported https://lichess.org/@/grantslatton/imported

gamegoblin 3y ago
Lots of those are various experiments with different prompting strategies, models, etc, so not representative. It regularly beats Stockfish Level 4 even without opening blunders. source: I am OP