6 ms·

Finding surprising moves in chess games

135 points by mdp 10y ago

Matetricks 10y ago
If I understand your code correctly in analyse_evaluations, you're defining a "surprising move" as a move that has a large change in valuation when it's considered at a higher depth. So if a "human" (really Stockfish at depth 5) evaluates a move as +1 and a "computer" (Stockfish now at depth 11) evaluates the move as +5, the move is surprising. This is pretty interesting, but I'm not sure if it fully captures all the nuances of what a surprising move is. You might be able to classify a move as tactically surprising if it becomes clear after depth 7 that the ending position is favorable. However, in my opinion truly surprising moves are ones that carry plans that I haven't even considered. Hence, this methodology doesn't capture moves that are positionally surprising as there wouldn't be such a drastic change in evaluation at different depths. I'm not sure where you would start to figure that one out though :) That being said this is really cool work!
- te_platt 10y ago
  Surprising may be a bit subjective. I know that I am all too often surprised by my opponent - and not in a good way. This may be a good way to study what kind of patterns have interesting weaknesses.
  - twothamendment 10y ago
    Surprising is very subjective. I was playing chess in a cabin full of people and found the checkers game next to me to be more entertaining - most likely because of the people. My chess moves were not quite random, but because I wasn't really paying attention they were really frustrating my opponent because he couldn't make sense of what I was doing. My moves were either not very logical or so brilliant that he didn't know what I was up to and it was really getting into his head. Surprising moves? Yes. Good moves? Not so much.
- yakult 10y ago
  You can filter for this: Take a database of games from pro players; Take the set of all moves where Stockfish-5 agrees that the move actually played is the optimum. Filter for all the moves where Stockfish-11 has a different opinion that results in a big gain in position. What you get is a list of moves that would surprise pro-players under time pressure. I wouldn't be surprised if professional chess players are all running a version of this against individual known opponents before a tournament to probe for weaknesses. A harder problem would be to cross-reference this final list with the post-game opinions published by professional commentators and identify major discrepancies. This would be the "wouldn't have thought of it in a million years" list.
  - Matetricks 10y ago
    The list that's returned still contains mainly tactical surprises where Stockfish inaccurately evaluated the position at the end of depth 5. I think what I'm trying to say is there are some moves in a position that aren't tactically surprising (a piece sacrifice, a crazy attacking move, etc.) but positionally surprising (a long maneuver to get a piece to a certain square that I didn't think of). These positionally surprising moves aren't captured by this methodology because they don't involve large fluctuations in valuation when the depth changes. As to your second point, an issue with how computer chess affects the modern scene is how playing the "best" move in any given position isn't representative of how humans play. Humans carry out plans and evaluate positions to the best of their ability, but the heuristics and procedure they use aren't the same as a computer's. For example, Karjakin didn't prepare for his match against Carlsen last month by playing a bunch of games against Stockfish. Rather he probably analyzed Carlsen's past games and opening choices to come up with a strategy. I do think you can come up with a way to prepare against individually known opponents by identifying weaknesses programmatically. You can model a human's approach to playing chess as a distribution of parameters (material, king safety, pawn structure, etc.) that take in the current position and return the best move. You also have Stockfish's evaluation which returns the "best" move. With this, it's possible that you could use build a neural network that learns to play very similarly to a certain player by using their past games as a training set and comparing the chosen move to Stockfish's move. The network could learn to mimic the heuristics that the human individual uses to make decisions and playing against this new AI would be great practice for preparing against specific opponents.
- zwischenzug 10y ago
  This is pretty much what I wanted to say. What is a surprising move varies greatly from player to machine. Here's a good example: http://www.chessgames.com/perl/chessgame?gid=1064780 http://www.chessgames.com/perl/chessgame?gid=1064780 Capa's move 10 here (Bd7) is completely surprising to the vast majority of players and computers. It breaks most of the standard 'rules' of development and space control. However, it doesn't move the needle in terms of tactical significance at all. To me, that's a surprising move.
vinchuco 10y ago
I wish it had examples! I'd be more interested in a move that complicates the game. In fact, what is the most complicated position in chess? (Legal or not)
- nimchimpsky 10y ago
  the starting position ?
  - deleted 10y ago
    [deleted]
  - vinchuco 10y ago
    No, add a pawn.
    - nimchimpsky 10y ago
      what ? You mean have a non-legal starting position ? Well yeah, I could also add a rubiks cube to the white pieces.
      - vinchuco 10y ago
        Fair point. But we'll eventually settle on some rules. The thought is like the following: people actually compose chess puzzles for fun, without regard to them appearing in an actual game: There's chess puzzle composition competitions. Could one write a program to make a monster puzzle (or better, the most complicated) with forced moves? That is: maximize X in "white to move and mate in X moves" restricted to board size and standard chess rules.
- vinchuco 10y ago
  My wish was granted!
- Isofarro 10y ago
  Probably positions stemming from the Botvinnik variation of the Semi-slav. Conventional chess wisdom doesn't really help, and figuring a way forward is pure calculation. Then also, maybe lines of the "Irrational system" of the French Defence, Winawer variation (lines of Qg4, and Black sacrificing both g7 and h7 pawns). Playing with nuances about what "complicated" means, perhaps also Nimzowitsch's "Immortal Zugzwang" game, where White is absolutely helpless on an almost full board of pieces.
pk2200 10y ago
You might also add the condition that best-move(deep-search) != best-move(shallow-search). IMO this is what really makes a move surprising - it looks bad, but is good. Btw, I've never seen Rxd4!! from Kasparov-Topalov described as a "natural" move. I'm guessing the author meant "unnatural". :)
- azag0 10y ago
  I read <irony>.
- zodiac 10y ago
  I think it's a bit of humor
al_biglan 10y ago
This seems a perfect candidate to test with some well known test games. Fischer's 17 ... Be6, for example in the Game of the Century Or 15 ...Nf2 (again Fischer and Byrne) http://www.chessgames.com/perl/chessgame?gid=1008419&kpage=1 http://www.chessgames.com/perl/chessgame?gid=1008419&kpage=1 It might be interesting to see how many blunders this identifies compared with traditional analysis. (Looking at the "surprising bad move" vs "surprising good move") In any case, very interesting work. Others have some good ideas for additional checks/considerations, and it would be interesting to see how this evolves!
- zzazzdsa 10y ago
  A good game to test this on is http://www.chessgames.com/perl/chessgame?gid=1139126 http://www.chessgames.com/perl/chessgame?gid=1139126 . Tal played the move 19. Rf6!!, which instantly wins the game. However, (at least when I tested this a few months ago), Stockfish takes a fairly long time to recognize this-- it prefers the more conservative 19. c4 instead.
  - Rainymood 10y ago
    I just went through the game and man 19. Rf6!! Took me by surprise! What an amazing move! I also would love to see some chess engine that plays ridiculous moves in the beginning to throw people off guard (i.e. increase greatly increase the variance in possible evaluations) and then crushes the opponent by playing all the computer moves while the human players have to walk on their toes in a minefield of possible blunders because the position is so wacky.
dsjoerg 10y ago
Really cool idea and interesting visualization! I am working on similar / related stuff if you'd like to collaborate further. dsjoerg at gmail.com. I participated in https://www.kaggle.com/c/finding-elo https://www.kaggle.com/c/finding-elo
schoen 10y ago
Note that Stockfish, which this project uses, just won the TCEC computer chess championship. https://en.wikipedia.org/wiki/TCEC https://en.wikipedia.org/wiki/TCEC There are other championships out there, but this is an indication that Stockfish is among the very strongest chess-playing entities in the world.
deleted 10y ago
[deleted]
billforsternz 10y ago
I've been thinking about adding a game-analyse feature to my Tarrasch Chess GUI https://triplehappy.com https://triplehappy.com. I was toying with the idea of using stockfish plus a poor engine as opposed to stockfish deep plus stockfish shallow. I was actually going to use my own Tarrasch Toy engine (about 1500 elo) as a reasonable proxy of a poor human player. My engine has fixed depth, so when set to very low depth will routinely just do the obvious thing and fall into simple fork traps etc., which would make it possible to auto generate simple explanatory variations like "Not $bad_move_played_by_my_engine because of $refutation_variation_pointed_out_by_stockfish." Like you I've noticed that often when looking at Engine Analysis the obvious human move doesn't appear as one of the first choices, which usually means there is an opportunity to get the machine to show you some tactics. So essentially your idea modulated by my assumption it would be difficult to get Stockfish to play sufficiently badly for stockfish deep plus stockfish shallow. But anyway, congrats to you for actually getting there first and making the idea work.
CYHSM 10y ago
Hi! I'm the author of this little library. Somebody apparently took it from my reddit thread (https://www.reddit.com/r/chess/comments/5i54lu/how_to_quantify_surprising_chess_moves/ https://www.reddit.com/r/chess/comments/5i54lu/how_to_quanti...) and linked it here. Thank you for doing that, i haven't thought this would interest so many people! I hope i can answer some of your questions given more time but I have to say there are some very cool suggestions here and in the reddit thread which i have to keep in mind.

135 points by mdp 10y ago

If I understand your code correctly in analyse_evaluations, you're defining a "surprising move" as a move that has a large change in valuation when it's considered at a higher depth. So if a "human" (really Stockfish at depth 5) evaluates a move as +1 and a "computer" (Stockfish now at depth 11) evaluates the move as +5, the move is surprising. This is pretty interesting, but I'm not sure if it fully captures all the nuances of what a surprising move is. You might be able to classify a move as tactically surprising if it becomes clear after depth 7 that the ending position is favorable. However, in my opinion truly surprising moves are ones that carry plans that I haven't even considered. Hence, this methodology doesn't capture moves that are positionally surprising as there wouldn't be such a drastic change in evaluation at different depths. I'm not sure where you would start to figure that one out though :) That being said this is really cool work!

te_platt 10y ago
Surprising may be a bit subjective. I know that I am all too often surprised by my opponent - and not in a good way. This may be a good way to study what kind of patterns have interesting weaknesses.
- twothamendment 10y ago
  Surprising is very subjective. I was playing chess in a cabin full of people and found the checkers game next to me to be more entertaining - most likely because of the people. My chess moves were not quite random, but because I wasn't really paying attention they were really frustrating my opponent because he couldn't make sense of what I was doing. My moves were either not very logical or so brilliant that he didn't know what I was up to and it was really getting into his head. Surprising moves? Yes. Good moves? Not so much.
yakult 10y ago
You can filter for this: Take a database of games from pro players; Take the set of all moves where Stockfish-5 agrees that the move actually played is the optimum. Filter for all the moves where Stockfish-11 has a different opinion that results in a big gain in position. What you get is a list of moves that would surprise pro-players under time pressure. I wouldn't be surprised if professional chess players are all running a version of this against individual known opponents before a tournament to probe for weaknesses. A harder problem would be to cross-reference this final list with the post-game opinions published by professional commentators and identify major discrepancies. This would be the "wouldn't have thought of it in a million years" list.
- Matetricks 10y ago
  The list that's returned still contains mainly tactical surprises where Stockfish inaccurately evaluated the position at the end of depth 5. I think what I'm trying to say is there are some moves in a position that aren't tactically surprising (a piece sacrifice, a crazy attacking move, etc.) but positionally surprising (a long maneuver to get a piece to a certain square that I didn't think of). These positionally surprising moves aren't captured by this methodology because they don't involve large fluctuations in valuation when the depth changes. As to your second point, an issue with how computer chess affects the modern scene is how playing the "best" move in any given position isn't representative of how humans play. Humans carry out plans and evaluate positions to the best of their ability, but the heuristics and procedure they use aren't the same as a computer's. For example, Karjakin didn't prepare for his match against Carlsen last month by playing a bunch of games against Stockfish. Rather he probably analyzed Carlsen's past games and opening choices to come up with a strategy. I do think you can come up with a way to prepare against individually known opponents by identifying weaknesses programmatically. You can model a human's approach to playing chess as a distribution of parameters (material, king safety, pawn structure, etc.) that take in the current position and return the best move. You also have Stockfish's evaluation which returns the "best" move. With this, it's possible that you could use build a neural network that learns to play very similarly to a certain player by using their past games as a training set and comparing the chosen move to Stockfish's move. The network could learn to mimic the heuristics that the human individual uses to make decisions and playing against this new AI would be great practice for preparing against specific opponents.
zwischenzug 10y ago
This is pretty much what I wanted to say. What is a surprising move varies greatly from player to machine. Here's a good example: http://www.chessgames.com/perl/chessgame?gid=1064780 http://www.chessgames.com/perl/chessgame?gid=1064780 Capa's move 10 here (Bd7) is completely surprising to the vast majority of players and computers. It breaks most of the standard 'rules' of development and space control. However, it doesn't move the needle in terms of tactical significance at all. To me, that's a surprising move.

vinchuco 10y ago

I wish it had examples! I'd be more interested in a move that complicates the game. In fact, what is the most complicated position in chess? (Legal or not)

nimchimpsky 10y ago
the starting position ?
- deleted 10y ago
  [deleted]
- vinchuco 10y ago
  No, add a pawn.
  - nimchimpsky 10y ago
    what ? You mean have a non-legal starting position ? Well yeah, I could also add a rubiks cube to the white pieces.
    - vinchuco 10y ago
      Fair point. But we'll eventually settle on some rules. The thought is like the following: people actually compose chess puzzles for fun, without regard to them appearing in an actual game: There's chess puzzle composition competitions. Could one write a program to make a monster puzzle (or better, the most complicated) with forced moves? That is: maximize X in "white to move and mate in X moves" restricted to board size and standard chess rules.
vinchuco 10y ago
My wish was granted!
Isofarro 10y ago
Probably positions stemming from the Botvinnik variation of the Semi-slav. Conventional chess wisdom doesn't really help, and figuring a way forward is pure calculation. Then also, maybe lines of the "Irrational system" of the French Defence, Winawer variation (lines of Qg4, and Black sacrificing both g7 and h7 pawns). Playing with nuances about what "complicated" means, perhaps also Nimzowitsch's "Immortal Zugzwang" game, where White is absolutely helpless on an almost full board of pieces.

pk2200 10y ago

You might also add the condition that best-move(deep-search) != best-move(shallow-search). IMO this is what really makes a move surprising - it looks bad, but is good. Btw, I've never seen Rxd4!! from Kasparov-Topalov described as a "natural" move. I'm guessing the author meant "unnatural". :)

azag0 10y ago
I read <irony>.
zodiac 10y ago
I think it's a bit of humor

al_biglan 10y ago

This seems a perfect candidate to test with some well known test games. Fischer's 17 ... Be6, for example in the Game of the Century Or 15 ...Nf2 (again Fischer and Byrne) http://www.chessgames.com/perl/chessgame?gid=1008419&kpage=1 http://www.chessgames.com/perl/chessgame?gid=1008419&kpage=1 It might be interesting to see how many blunders this identifies compared with traditional analysis. (Looking at the "surprising bad move" vs "surprising good move") In any case, very interesting work. Others have some good ideas for additional checks/considerations, and it would be interesting to see how this evolves!

zzazzdsa 10y ago
A good game to test this on is http://www.chessgames.com/perl/chessgame?gid=1139126 http://www.chessgames.com/perl/chessgame?gid=1139126 . Tal played the move 19. Rf6!!, which instantly wins the game. However, (at least when I tested this a few months ago), Stockfish takes a fairly long time to recognize this-- it prefers the more conservative 19. c4 instead.
- Rainymood 10y ago
  I just went through the game and man 19. Rf6!! Took me by surprise! What an amazing move! I also would love to see some chess engine that plays ridiculous moves in the beginning to throw people off guard (i.e. increase greatly increase the variance in possible evaluations) and then crushes the opponent by playing all the computer moves while the human players have to walk on their toes in a minefield of possible blunders because the position is so wacky.

dsjoerg 10y ago

Really cool idea and interesting visualization! I am working on similar / related stuff if you'd like to collaborate further. dsjoerg at gmail.com. I participated in https://www.kaggle.com/c/finding-elo https://www.kaggle.com/c/finding-elo

schoen 10y ago

Note that Stockfish, which this project uses, just won the TCEC computer chess championship. https://en.wikipedia.org/wiki/TCEC https://en.wikipedia.org/wiki/TCEC There are other championships out there, but this is an indication that Stockfish is among the very strongest chess-playing entities in the world.

deleted 10y ago

[deleted]

billforsternz 10y ago

I've been thinking about adding a game-analyse feature to my Tarrasch Chess GUI https://triplehappy.com https://triplehappy.com. I was toying with the idea of using stockfish plus a poor engine as opposed to stockfish deep plus stockfish shallow. I was actually going to use my own Tarrasch Toy engine (about 1500 elo) as a reasonable proxy of a poor human player. My engine has fixed depth, so when set to very low depth will routinely just do the obvious thing and fall into simple fork traps etc., which would make it possible to auto generate simple explanatory variations like "Not $bad_move_played_by_my_engine because of $refutation_variation_pointed_out_by_stockfish." Like you I've noticed that often when looking at Engine Analysis the obvious human move doesn't appear as one of the first choices, which usually means there is an opportunity to get the machine to show you some tactics. So essentially your idea modulated by my assumption it would be difficult to get Stockfish to play sufficiently badly for stockfish deep plus stockfish shallow. But anyway, congrats to you for actually getting there first and making the idea work.

CYHSM 10y ago

Hi! I'm the author of this little library. Somebody apparently took it from my reddit thread (https://www.reddit.com/r/chess/comments/5i54lu/how_to_quantify_surprising_chess_moves/ https://www.reddit.com/r/chess/comments/5i54lu/how_to_quanti...) and linked it here. Thank you for doing that, i haven't thought this would interest so many people! I hope i can answer some of your questions given more time but I have to say there are some very cool suggestions here and in the reddit thread which i have to keep in mind.