21 ms·

Positions chess engines don't understand

407 points by diplodocusaur 5y ago

TchoBeer 5y ago
I think there are some engines (Crystal is the one I'm thinking of) which do well in fortresses; these come at the cost of play strength.
- thaumasiotes 5y ago
  Presumably the next step is a synthesizer which can choose the appropriate engine to delegate to based on its own reading of the board.
  - TchoBeer 5y ago
    Fortresses and the types of closed positions that confuse engines are exceedingly rare. The benefit of fortress detection is necessarily overtaken by the runtime cost.
  - deleted 5y ago
    [deleted]
senkora 5y ago
It seems dubious to show that engines are sometimes bad at evaluating positions by giving a position with three black bishops on black squares.
- lmilcin 5y ago
  While unheard of, this is not illegal. You could theoretically convert two pawns to bishops and have three black bishops. Nobody would do that as it is usual to convert pawns to queens, but it is within the rules for you to choose. So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.
  - ioseph 5y ago
    I could imagine such a move making sense with the king on the backline such that promotion to queen would trigger a stalemate
    - johannes1234321 5y ago
      In rare cases this might happen, two times in a match very unlikely. The only situation where this might be worthwhile would be a forced mate in consequence. Otherwise the Queen is worthy enough that you can afford the tempo loss.
    - tzs 5y ago
      If it is in a tournament, it could even happen for reasons not related to the position itself. Let's say you have a winning position, but you are in severe time pressure. You really need a few moments to stop and think to figure out possible stalemate swindles your opponent might have. You are about to promote to a queen. You realize that both of your bishops are still on the board, and promote to a bishop instead of a queen. That lets you stop the clock and summon an arbiter to go find another bishop of the right color. While the arbiter if off dealing with that, you get your much needed thinking time.
    - deleted 5y ago
      [deleted]
  - tialaramex 5y ago
    Underpromotions are one of those "toy problem" things. A human might finish a whole professional career having never once promoted a pawn to anything but a queen, but in theoretical problem positions - like the one you're talking about - they happen "all the time" because it's a fun twist. So it's understandable for a machine not to even bother modelling these weird cases I think.
    - perihelions 5y ago
      edit: This comment is wrong and useless; I'm leaving it up to preserve the logic of the replies. To quantify, here's raw counts from a representative sample of serious games: $ rg -Io '=\w' twic*.pgn | sort | uniq -c 365 =B 1551 =N 122386 =Q 1393 =R I see knight (N) under-promotions pretty regularly on chess streams. The common pattern is there isn't a free tempo to promote the pawn, unless promoting the pawn creates an immediate threat that holds the attacker's initiative; and a knight at that particular moment creates that opportunity.
- awb 5y ago
  There are other more realistic examples like the Nakamura game sacrificing 2 useless rooks to give the engine a false sense of winning. Then the engine sacrifices a valuable pawn to avoid a 50 move draw. Turn out the pawn was much more valuable than the 2 rooks in the particular closed position and Nakamura goes on to win. Basically, the only way to beat an engine is to try to create an unusual situation that the AI hasn’t practiced before. But as others other out, AI suffers when it doesn’t have enough experience in a particular situation. So, the author is really just pointing out the extreme edge cases AI hasn’t mastered yet. But over time and getting these examples into the training process there’s no reason to believe that the AI couldn’t learn these situations as well.
  - anotherqqq 5y ago
    Once you tell a human 'there's a tactic, go', given enough time we find it.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- contravariant 5y ago
  Is it though? If humans instantly realise that the three black bishops have some special significance and a chess engine does not then apparently the chess engine lacks some understanding that humans do have.
  - vkou 5y ago
    Sure, but it's 'meta' knowledge. If, for instance, the name of a chess problem includes the solution in it, written in plain English, no current chess engine is going to pick up on this bit of 'meta' information - yet any English-speaking human trying to solve the problem would be trivially able to use it. Does that make humans 'better' at chess than chess engines? Not really. Does it make humans 'better' at solving a given chess position, given some metacontextual clues? It does, but it is a bit contrived.
    - contravariant 5y ago
      It's not exactly 'meta' knowledge, it's something that anyone with a few hours of experience in chess would be able to pick up on. It doesn't require knowledge of anything other than chess. If an AI can't pick up on it then that suggests it lacks parts of what to humans would be a basic understanding of chess.
    - taneq 5y ago
      It does, however, seem to be the ‘secret sauce’ to humans’ ability to generalise our problem solving across domains.
    - akyu 5y ago
      It's not 'meta' knowledge, it's a basic rule of the game.
      - vkou 5y ago
        Promoting to a bishop is a rule of the game. Solving a chess puzzle where there are for some reason three black-square bishops - recognizing that the bishops are important to the puzzle is meta-knowledge.
matsemann 5y ago
When watching the WC games, I've seen it happen that a move wasn't considered as a top move by the engine, but once played the engine realizes it's actually crushing. Something about the heuristics used to prune the vast search space can make it miss sacrifices or seemingly sub-optimal moves that temporarily weakens the perceived position but has a huge payoff in the end. But humans find them. Of course, given enough time and depth the engine will eventually circle back and try the move. But it has no intuition. Also, an engine without an endgame tablebase can be pretty stupid. There are certain rules one can deduct when there are few pieces left, but a min/max engine will search forever, not knowing the patterns.
- thaumasiotes 5y ago
  > I've seen it happen that a move wasn't considered as a top move by the engine, but once played the engine realizes it's actually crushing. Something about the heuristics used to prune the vast search space can make it miss sacrifices or seemingly sub-optimal moves that temporarily weakens the perceived position but has a huge payoff in the end. But humans find them. Just to observe, humans display exactly the same phenomenon of ignoring a move before it's made while still being able to realize, after it's made, that it was very strong and ignoring it was a mistake.
  - curiousllama 5y ago
    The word I would use for a move like that is “counterintuitive” Which... suggests the computer displays intuition
    - taneq 5y ago
      Exactly. In fact you could argue that intuition, ‘arriving at a conclusion without a conscious understanding of your reasoning’, is the only thing a neural net displays.
    - dalbasal 5y ago
      Well played. OP's choice of words is inexact. The engine has a different intuition, not none. Humans will usually explore trades and overlook seemingly "slow" moves that the computer usually finds. What the engine does lacks is what chess players call "theory," hence the need for endgame DBs. Humans can also beat engines with "logic," like the Naka match. That one was extraordinary. A simpler & more typical example is isolating a pawn so that it can't ever be defended, but also can't be attacked immediately. For a human, it's easy to understand this is long term vulnerability even if the pawn survives for 20+ more moves. Anti-computer strategies, ironically, force you to "play the player, not the board."
dwohnitmok 5y ago
Does anybody know if advanced chess/centaur chess (chess play where a human uses a computer for assistance) is still a thing/whether a human+computer combo is a meaningful improvement these days (i.e. last couple of years) over just a computer. I can't find any recent advanced chess tournaments and though I see quotes of people saying that the combo is stronger than a computer alone, I haven't found any recent examples of a top tier engine by itself losing to a human + engine (e.g. Stockfish + human vs Stockfish).
- edouard-harris 5y ago
  Fully automated engines are now probably even with centaur teams. See, for example, this great write-up: https://www.gwern.net/Notes#advanced-chess-obituary https://www.gwern.net/Notes#advanced-chess-obituary
  - dwohnitmok 5y ago
    A quick glance doesn't seem to give conclusive evidence that pure engine play strictly dominates centaurs (the footnotes only have tournaments where centaurs still win, but these tournaments are also getting a bit old). The usual messaging I see around centaur-based styles such as certain correspondence chess tournaments is that you will lose if you just do "push-button play," that is just blindly do what the computer tells you to do. I'm curious if that's no longer true with the new crop of ML engines.
    - edouard-harris 5y ago
      You're absolutely right! Edited the gp from "strictly dominate" to "are probably even with". My memory of the piece was playing tricks on me. That was a serious error, thanks for catching it. My best guess based on a rereading of the footnotes is that the performance ceiling for chess is probably low enough that it has been ~reached by both centaur teams and pure engines. So the two would have been operating neck-and-neck as of ~2017, with win rates largely determined by random-ish factors like human mis-clicks (also mentioned in the article).
perihelions 5y ago
Here's a particularly extreme example: https://old.reddit.com/r/chess/comments/ndz2lj/simple_mate_in_93/ https://old.reddit.com/r/chess/comments/ndz2lj/simple_mate_i... It's a mate-in-93 puzzle that's fairly accessible to humans, using abstract reasoning. But not chess engines. Comparing against the OP article, the main "technique"/"trick" is zugzwang (#7), but on a dramatic scale. I think this is the kind of position you could use to stress-test a candidate puzzle solver, just because of the shear size of the solution.
airza 5y ago
An astonishing insert: >A good example of human exploitation of the engine's failings is GM Hikaru Nakamura's defeat of Rybka in the following three-minute blitz game from the Internet Chess Club. Nakamura cleverly locks the position so that progress is impossible for Rybka, then he offers two exchange sacrifices to the engine. With the position locked, the engine's rooks have no value, but the engine thinks it has a winning material advantage. With a draw by the 50-move rule approaching, the engine sacrifices a pawn to avoid a draw, but this proves a huge mistake as Nakamura is then able to win the game, an incredible achievement!
- someguy101010 5y ago
  There is a video of this on YouTube https://youtu.be/DTMnrUdd8Tw https://youtu.be/DTMnrUdd8Tw
  - Clewza313 5y ago
    Thanks for the link. The commentary notes that this was possible because Rybka is an algorithmic engine with hard constraints like "must try to win in 50 moves to avoid tie", and this is what Nakamura exploited to win. DeepMind and other deep learning/neural network approaches don't have hard rules like this.
    - gpm 5y ago
      > because Rybka is an algorithmic engine with hard constraints like "must try to win in 50 moves to avoid tie" That rule isn't the engine's constraint, that's a rule of chess. 50 moves without a pawn move or piece capture (basically "irrevocable progression of the game") is a draw. The chess engine thought it was winning, so it wanted to avoid a draw, even if it meant putting itself in a still winning but otherwise worse situation. Unfortunately for the chess engine it was mistaken, in the original position it wasn't winning, but drawing, and in the new position it wasn't still winning, but losing.
      - nvader 5y ago
gweinberg 5y ago
Has anyone actually checked that a modern chess engine believes black is winning in the Penrose position? I find it very hard to believe.
- shric 5y ago
  Stockfish 12 (280920) on my phone rates black as up 19.71
- perihelions 5y ago
  Lichess cloud compute has that position. It still thinks black is winning at depth 59 (Stockfish 11 HCE). https://lichess.org/analysis/8/p7/kpP5/qrp1b3/rpP2b2/pP4b1/P3K3/8_w_-_-_0_1 https://lichess.org/analysis/8/p7/kpP5/qrp1b3/rpP2b2/pP4b1/P...
  - shric 5y ago
    Indeed, and same with Stockfish 13 running locally.
- dwohnitmok 5y ago
  It looks like Stockfish still thinks black is winning. I'd be curious to see if Leela Chess Zero also thinks black is winning.
  - gweinberg 5y ago
    Is that 59 ply? 59 moves without a pawn move or capture means the game was drawn nine moves ago. Here's what's confusing me: if the chess engine has an initial numeric value for the current position, and the positions going out to the end of its search all have the same value or worse, shouldn't it be pretty trivial to conclude it is not making progress?
    - shric 5y ago
      Yes, I guess Stockfish doesn't bother to consider a draw on the 50 move rule. It does, iirc, go for a draw if it has no better via threefold repetition.
sillysaurusx 5y ago
When OpenAI trained their Dota model to beat people 1v1 mid SF, they had a contest at The International (the big yearly dota tournament, RIP) like "can you beat OpenAI's new bot?" and had a big bucket of prizes for anyone that could. By the end of the night, the bucket was empty. People learned to cheese the bot by running "out of bounds", so to speak -- normally in a 1v1, you're supposed to stay close to your opponent, since they'll be getting stronger if you leave. But the bot didn't know how to deal with it when you snuck behind it (normally an insane maneuver, almost guaranteed to cost you the game in normal play) and prevented his army (called "creeps") from running to the middle lane. (All you have to do is go wave your hand at them, and they mindlessly chase you.) People did that over and over, and your own army would eventually overwhelm the bot and win. :)
- inostia 5y ago
  That's a great story! It definitely speaks to human beings' knack for finding edge cases. Do you know what year that was? I'm wondering if AI has advanced and patched those weaknesses.
  - sillysaurusx 5y ago
    I am of the surprisingly-controversial opinion that ML’s future is going to be as a human empowerment tool — a bicycle for the mind, yet so much more. Listen to how cool my gamedev music sounds: https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... I feel comfortable saying that it’s awesome and being immodest, because I didn’t write it! None of those notes are mine. “Crossing the Channel” has one of the strangest tempos I’ve heard, yet it sounds so cool because I think the model made a mistake early on, and decided it wasn’t a mistake — instead, it came up with the most likely not-mistake it could think of, and it happened to sound great. But I “wrote” all ~20 songs on that list in one night. Took about six hours, aka a standard workday. I don’t think it’s production-grade music - far from it - but it also wasn’t a production grade model. (On the other hand, it was trained by gwern, so I’m skeptical the production models of the future will capture all the little magic that he seems to imbue into his.) I chose the instruments, I decided what sounded good, but I didn’t write a single note. It felt a lot like listening to someone jam out, and asking them to play different things. I was just the fella who was there to listen. That’s where ML is going to shine. The future is going to be so exciting with the things you’ll be able to do. But not in the current direction, I think. Currently, the goal is to factor the human out of the equation entirely. So the answer to your question is: even if the AI has “advanced”, its only because they added this case to the training set. If a human was at the helm, they would have annihilated you, because they’d see what you were doing and pilot the AI over to you. I am skeptical this case is solvable, in a fully automatic way. IMO human empowerment will happen by arming actual humans with these models. (Depressingly, “arming” might have a few meanings depending on how AI war turns out, but that’s a different conversation.)
rudi-c 5y ago
There's a similar situation with Go where some positions utterly confuse bots trained on playing mostly normal games. There was this interesting research blog post on training a bot specifically to become good at solving one of these weird problems (Igo Hatsuyoron 120, the "hardest go problem ever") https://blog.janestreet.com/deep-learning-the-hardest-go-problem-in-the-world/ https://blog.janestreet.com/deep-learning-the-hardest-go-pro...
- kadoban 5y ago
  It is worth noting that that problem is only understood by humans (as well as it is) after _centuries_ of study by many professional players. My understanding is that even very strong human players must study that problem for days/months/years to really understand it well. So katago being unable to handle it without special training doesn't seem _quite_ as blind of a blindspot as the chess examples from the article seem to be (I suck at chess and I was able to understand one or two). I'm not trying to undermine you mentioning this, in case it comes off like that, on the contrary I think the comparison is quite interesting. I'm curious if this is just a difference in go vs chess, or in the relative abilities of specific kinds of AIs to handle these, or maybe just differences in human ability to craft and/or understand different problem difficulties between the games.
unnouinceput 5y ago
Quote: "Since IBM's Deep Blue defeated World Chess Champion Garry Kasparov in their 1997 match..." Deep Blue lost in 1996. Its upgrade, called Deeper Blue is the one that won the famous match in 1997. Please SamCopland, do your homework.
- shric 5y ago
  Do you have a source? Deeper Blue was only ever an unofficial nickname. The upgraded Deep Blue was still Deep Blue.
  - unnouinceput 5y ago
    Per Wikipedia "heavy upgrade". Same like google of today is the same name as the one launched at beginning of 00's. Do you really think that from technological perspective is the same search engine or a heavy upgrade? The only thing Deep Blue and Deeper Blue have in common is that they were both put in the same case and created by IBM, everything else is different
johnklos 5y ago
Nice try, chess.com. I'm not going to let you use all those cycles stolen by your site to win chess games :P
vmception 5y ago
Yeah I would have thought these engines to be alot smarter then this, especially the “AI” ones
microtherion 5y ago
The Hasek study in #2 is bugging me: Why can't Black simply play 1. ... Rh8 ? It looks to me like this would gain the one crucial tempo to win the game. None of the discussion of the study I've found seems to consider this move (And the online analysis engines are useless, as they don't properly understand that the original line is a draw, as the article notes). ETA: Corrected typo, I originally wrote 1. ... Re8, which does not accomplish anything.
- trykondev 5y ago
  I think white just continues moving the king toward the center -- if black plays waiting moves, white can bring the king to c1-d1-e1 and then shuffle the king between e1 and f1. Eventually white will sacrifice the rook in the same pattern as the main line, and then play Ke2 -- there's no "wrong" square for the white king to be stuck on such that Ke2 would be unreachable, so black doesn't gain anything from the extra waiting moves.
  - microtherion 5y ago
    Ah, sorry, I had a typo, I meant 1. ... Rh8, after which the rook sacrifice does not quite work the same way anymore. I agree that 1. ... Re8 does not help.
    - trykondev 5y ago
      Ah, understood -- I spent some time trying to figure it out after seeing your clarification. You are right that if you try the same rook sac technique, white won't be able to get the white king over in time to defend g2. It turns out there is another very impressive move white needs to spot: after 1...Rh8, white needs to find 2. Rf8! and now black has to either trade into the drawn king and pawn ending, or take the rook and give back the tempo that's just been saved. Now the white king is in time again. Quite a move, that should be listed as a sideline in the puzzle for sure!
      - microtherion 5y ago
deleted 5y ago
[deleted]
Andrex 5y ago
https://www.youtube.com/watch?v=Z4F7mUUjt_c https://www.youtube.com/watch?v=Z4F7mUUjt_c Maybe a more realistic scene than the fanbase gives it credit for. :)
- aidenn0 5y ago
  If it weren't a well-studied gambit perhaps...
zone411 5y ago
So I actually checked the problems listed. In the "IQ Test #52" position (FEN: 8/1p1q1k2/1Pp5/p1Pp4/P2Pp1p1/4PpPp/1N3P1P/3B2K1 w - - 0 1) listed in #1 both LC0 and Stockfish play the correct line on my computer in seconds. Both of them also play the right move Ba4+ in the second position of #1 "William Rudolph vs." (FEN: 8/1p1q1k2/1Pp5/p1Pp4/P2Pp1p1/4PpPp/1N3P1P/3B2K1 w - - 0 1) but they take quite a bit longer. Stockfish variants get it quicker. Stockfish solves "Hasek vs." (FEN: r7/7k/5R2/p3p3/Pp1pPp2/1PpP1Pp1/K1P3P1/8 w - - 0 1) listed in #2 quickly. Both Stockfish and LC0 solve "Lazard=F vs." (FEN: q7/8/2p5/B2p2pp/5pp1/2N3k1/6P1/7K w - - 0 1) quickly. Stockfish gets Bh3 from #3 "Veselin Topalov (?) vs. Alexey Shirov" (FEN: 8/8/4kpp1/3p1b2/p6P/2B5/6P1/6K1 b - - 2 47) in seconds (7-piece end game tablebases installed). The next position from #3 "Spassky, Boris V vs. Byrne, R." (FEN: 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P4P/PPB1K3/8 w - - 0 1) is easy for both Stockfish and LC0. They both get 50. c5!! right away. Stockfish also gets the last position from #3 "Stefan Brzozka vs. David Bronstein" in seconds (FEN: 1r6/4k3/r2p2p1/2pR1p1p/2P1pP1P/pPK1P1P1/P7/1B6 b - - 0 48) Rxb3+. Stockfish and LC0 see Kd1 in "Lamford=P vs." from #4 (FEN: 8/8/8/1k3p2/p1p1pPp1/PpPpP1Pp/1P1P3P/QNK2NRR w - - 0 1) but believe Rg2 is also winning. Stockfish gets c8N from "Randviir=J vs." (FEN: 5nr1/2Pp2pk/3Pp1p1/4P1P1/6P1/5K2/8/7n w - - 0 1) in #4 in about 2 minutes on my computer. Stockfish gets Bf5 from "Simkhovich=F vs." (FEN: 8/8/2pk4/8/p1p3B1/PpP5/1P6/r1NK4 w - - 2 2) in #5 in seconds. LC0 also gets it. The next two positions are mentioned as easier for programs and they are: Qe3 from "Deep Blue vs. Garry Kasparov" in #6 (FEN: 1r6/5kp1/RqQb1p1p/1p1PpP2/1Pp1B3/2P4P/6P1/5K2 b - - 14 45) is very easy for both Stockfish and LC0. Both also get "Vladimir Kramnik vs. Peter Leko" (FEN: 6k1/5p1p/P1pb1nq1/6p1/3P4/1BP2PP1/1P1Nb2P/R1B3K1 b - - 0 25) in #6 quickly. "Matous=M vs." (FEN: n2Bqk2/5p1p/Q4KP1/p7/8/8/8/8 w - - 0 1) is indeed harder for Stockfish and LC0 than expected. I've confirmed mate in 13 in my mate solver. In "Nigel Short vs. Vladimir Kramnik" (FEN: r3r1k1/1bp1Bppp/pb1p4/1p6/1P6/1BP2P2/P1P2PKP/R3R3 b - - 6 19) from #9, the engines like ...a5 more than ...c6. Hard to say that this move doesn't also win without more analysis. Stockfish wants to play c6! and b4! from "Marwitz=J vs." from #10 (FEN: 2K3k1/1p6/R3p1p1/1rB1P1P1/8/8/1Pb5/8 w - - 0 1) right from the start. LC0 takes longer but gets it as well. In "Anish Giri vs. Maxim Rodshtein" (FEN: 8/5pkp/3p1np1/Rpr5/8/6P1/PB3PKP/8 w - - 6 34) from #10 both Stockfish and LC0 like 34. h4 over 34. a4. More analysis would be needed to see if h4 is not also winning. Last position "IQ Test #16" (FEN: 5k2/4bp2/2B3p1/1P4p1/3R4/3P2PP/2r2PK1/8 b - - 0 1) takes around 10 seconds for Stockfish. In summary, chess engine might not really "understand" these positions but they solve them pretty well.
mrslave 5y ago
Agadmator covered Kramnik v Leko 2002 (in 2018) titled "Invisible to Engines | One Of The Greatest Moves Ever Played" [0] which is worth a watch. [0] https://www.youtube.com/watch?v=yGnpewUKP88 https://www.youtube.com/watch?v=yGnpewUKP88
ngcc_hk 5y ago
Politics are about making up rules that others have to follow. May be AI still not good at playing this level of politics.

407 points by diplodocusaur 5y ago

I think there are some engines (Crystal is the one I'm thinking of) which do well in fortresses; these come at the cost of play strength.

thaumasiotes 5y ago
Presumably the next step is a synthesizer which can choose the appropriate engine to delegate to based on its own reading of the board.
- TchoBeer 5y ago
  Fortresses and the types of closed positions that confuse engines are exceedingly rare. The benefit of fortress detection is necessarily overtaken by the runtime cost.
- deleted 5y ago
  [deleted]

senkora 5y ago

It seems dubious to show that engines are sometimes bad at evaluating positions by giving a position with three black bishops on black squares.

lmilcin 5y ago
While unheard of, this is not illegal. You could theoretically convert two pawns to bishops and have three black bishops. Nobody would do that as it is usual to convert pawns to queens, but it is within the rules for you to choose. So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.
- ioseph 5y ago
  I could imagine such a move making sense with the king on the backline such that promotion to queen would trigger a stalemate
  - johannes1234321 5y ago
    In rare cases this might happen, two times in a match very unlikely. The only situation where this might be worthwhile would be a forced mate in consequence. Otherwise the Queen is worthy enough that you can afford the tempo loss.
  - tzs 5y ago
    If it is in a tournament, it could even happen for reasons not related to the position itself. Let's say you have a winning position, but you are in severe time pressure. You really need a few moments to stop and think to figure out possible stalemate swindles your opponent might have. You are about to promote to a queen. You realize that both of your bishops are still on the board, and promote to a bishop instead of a queen. That lets you stop the clock and summon an arbiter to go find another bishop of the right color. While the arbiter if off dealing with that, you get your much needed thinking time.
  - deleted 5y ago
    [deleted]
- tialaramex 5y ago
  Underpromotions are one of those "toy problem" things. A human might finish a whole professional career having never once promoted a pawn to anything but a queen, but in theoretical problem positions - like the one you're talking about - they happen "all the time" because it's a fun twist. So it's understandable for a machine not to even bother modelling these weird cases I think.
  - perihelions 5y ago
    edit: This comment is wrong and useless; I'm leaving it up to preserve the logic of the replies. To quantify, here's raw counts from a representative sample of serious games: $ rg -Io '=\w' twic*.pgn | sort | uniq -c 365 =B 1551 =N 122386 =Q 1393 =R I see knight (N) under-promotions pretty regularly on chess streams. The common pattern is there isn't a free tempo to promote the pawn, unless promoting the pawn creates an immediate threat that holds the attacker's initiative; and a knight at that particular moment creates that opportunity.
awb 5y ago
There are other more realistic examples like the Nakamura game sacrificing 2 useless rooks to give the engine a false sense of winning. Then the engine sacrifices a valuable pawn to avoid a 50 move draw. Turn out the pawn was much more valuable than the 2 rooks in the particular closed position and Nakamura goes on to win. Basically, the only way to beat an engine is to try to create an unusual situation that the AI hasn’t practiced before. But as others other out, AI suffers when it doesn’t have enough experience in a particular situation. So, the author is really just pointing out the extreme edge cases AI hasn’t mastered yet. But over time and getting these examples into the training process there’s no reason to believe that the AI couldn’t learn these situations as well.
- anotherqqq 5y ago
  Once you tell a human 'there's a tactic, go', given enough time we find it.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
contravariant 5y ago
Is it though? If humans instantly realise that the three black bishops have some special significance and a chess engine does not then apparently the chess engine lacks some understanding that humans do have.
- vkou 5y ago
  Sure, but it's 'meta' knowledge. If, for instance, the name of a chess problem includes the solution in it, written in plain English, no current chess engine is going to pick up on this bit of 'meta' information - yet any English-speaking human trying to solve the problem would be trivially able to use it. Does that make humans 'better' at chess than chess engines? Not really. Does it make humans 'better' at solving a given chess position, given some metacontextual clues? It does, but it is a bit contrived.
  - contravariant 5y ago
    It's not exactly 'meta' knowledge, it's something that anyone with a few hours of experience in chess would be able to pick up on. It doesn't require knowledge of anything other than chess. If an AI can't pick up on it then that suggests it lacks parts of what to humans would be a basic understanding of chess.
  - taneq 5y ago
    It does, however, seem to be the ‘secret sauce’ to humans’ ability to generalise our problem solving across domains.
  - akyu 5y ago
    It's not 'meta' knowledge, it's a basic rule of the game.
    - vkou 5y ago
      Promoting to a bishop is a rule of the game. Solving a chess puzzle where there are for some reason three black-square bishops - recognizing that the bishops are important to the puzzle is meta-knowledge.

matsemann 5y ago

When watching the WC games, I've seen it happen that a move wasn't considered as a top move by the engine, but once played the engine realizes it's actually crushing. Something about the heuristics used to prune the vast search space can make it miss sacrifices or seemingly sub-optimal moves that temporarily weakens the perceived position but has a huge payoff in the end. But humans find them. Of course, given enough time and depth the engine will eventually circle back and try the move. But it has no intuition. Also, an engine without an endgame tablebase can be pretty stupid. There are certain rules one can deduct when there are few pieces left, but a min/max engine will search forever, not knowing the patterns.

thaumasiotes 5y ago
> I've seen it happen that a move wasn't considered as a top move by the engine, but once played the engine realizes it's actually crushing. Something about the heuristics used to prune the vast search space can make it miss sacrifices or seemingly sub-optimal moves that temporarily weakens the perceived position but has a huge payoff in the end. But humans find them. Just to observe, humans display exactly the same phenomenon of ignoring a move before it's made while still being able to realize, after it's made, that it was very strong and ignoring it was a mistake.
- curiousllama 5y ago
  The word I would use for a move like that is “counterintuitive” Which... suggests the computer displays intuition
  - taneq 5y ago
    Exactly. In fact you could argue that intuition, ‘arriving at a conclusion without a conscious understanding of your reasoning’, is the only thing a neural net displays.
  - dalbasal 5y ago
    Well played. OP's choice of words is inexact. The engine has a different intuition, not none. Humans will usually explore trades and overlook seemingly "slow" moves that the computer usually finds. What the engine does lacks is what chess players call "theory," hence the need for endgame DBs. Humans can also beat engines with "logic," like the Naka match. That one was extraordinary. A simpler & more typical example is isolating a pawn so that it can't ever be defended, but also can't be attacked immediately. For a human, it's easy to understand this is long term vulnerability even if the pawn survives for 20+ more moves. Anti-computer strategies, ironically, force you to "play the player, not the board."

dwohnitmok 5y ago

Does anybody know if advanced chess/centaur chess (chess play where a human uses a computer for assistance) is still a thing/whether a human+computer combo is a meaningful improvement these days (i.e. last couple of years) over just a computer. I can't find any recent advanced chess tournaments and though I see quotes of people saying that the combo is stronger than a computer alone, I haven't found any recent examples of a top tier engine by itself losing to a human + engine (e.g. Stockfish + human vs Stockfish).

edouard-harris 5y ago
Fully automated engines are now probably even with centaur teams. See, for example, this great write-up: https://www.gwern.net/Notes#advanced-chess-obituary https://www.gwern.net/Notes#advanced-chess-obituary
- dwohnitmok 5y ago
  A quick glance doesn't seem to give conclusive evidence that pure engine play strictly dominates centaurs (the footnotes only have tournaments where centaurs still win, but these tournaments are also getting a bit old). The usual messaging I see around centaur-based styles such as certain correspondence chess tournaments is that you will lose if you just do "push-button play," that is just blindly do what the computer tells you to do. I'm curious if that's no longer true with the new crop of ML engines.
  - edouard-harris 5y ago
    You're absolutely right! Edited the gp from "strictly dominate" to "are probably even with". My memory of the piece was playing tricks on me. That was a serious error, thanks for catching it. My best guess based on a rereading of the footnotes is that the performance ceiling for chess is probably low enough that it has been ~reached by both centaur teams and pure engines. So the two would have been operating neck-and-neck as of ~2017, with win rates largely determined by random-ish factors like human mis-clicks (also mentioned in the article).

perihelions 5y ago

Here's a particularly extreme example: https://old.reddit.com/r/chess/comments/ndz2lj/simple_mate_in_93/ https://old.reddit.com/r/chess/comments/ndz2lj/simple_mate_i... It's a mate-in-93 puzzle that's fairly accessible to humans, using abstract reasoning. But not chess engines. Comparing against the OP article, the main "technique"/"trick" is zugzwang (#7), but on a dramatic scale. I think this is the kind of position you could use to stress-test a candidate puzzle solver, just because of the shear size of the solution.

airza 5y ago

An astonishing insert: >A good example of human exploitation of the engine's failings is GM Hikaru Nakamura's defeat of Rybka in the following three-minute blitz game from the Internet Chess Club. Nakamura cleverly locks the position so that progress is impossible for Rybka, then he offers two exchange sacrifices to the engine. With the position locked, the engine's rooks have no value, but the engine thinks it has a winning material advantage. With a draw by the 50-move rule approaching, the engine sacrifices a pawn to avoid a draw, but this proves a huge mistake as Nakamura is then able to win the game, an incredible achievement!

someguy101010 5y ago
There is a video of this on YouTube https://youtu.be/DTMnrUdd8Tw https://youtu.be/DTMnrUdd8Tw
- Clewza313 5y ago
  Thanks for the link. The commentary notes that this was possible because Rybka is an algorithmic engine with hard constraints like "must try to win in 50 moves to avoid tie", and this is what Nakamura exploited to win. DeepMind and other deep learning/neural network approaches don't have hard rules like this.
  - gpm 5y ago
    > because Rybka is an algorithmic engine with hard constraints like "must try to win in 50 moves to avoid tie" That rule isn't the engine's constraint, that's a rule of chess. 50 moves without a pawn move or piece capture (basically "irrevocable progression of the game") is a draw. The chess engine thought it was winning, so it wanted to avoid a draw, even if it meant putting itself in a still winning but otherwise worse situation. Unfortunately for the chess engine it was mistaken, in the original position it wasn't winning, but drawing, and in the new position it wasn't still winning, but losing.
    - nvader 5y ago

gweinberg 5y ago

Has anyone actually checked that a modern chess engine believes black is winning in the Penrose position? I find it very hard to believe.

shric 5y ago
Stockfish 12 (280920) on my phone rates black as up 19.71
perihelions 5y ago
Lichess cloud compute has that position. It still thinks black is winning at depth 59 (Stockfish 11 HCE). https://lichess.org/analysis/8/p7/kpP5/qrp1b3/rpP2b2/pP4b1/P3K3/8_w_-_-_0_1 https://lichess.org/analysis/8/p7/kpP5/qrp1b3/rpP2b2/pP4b1/P...
- shric 5y ago
  Indeed, and same with Stockfish 13 running locally.
dwohnitmok 5y ago
It looks like Stockfish still thinks black is winning. I'd be curious to see if Leela Chess Zero also thinks black is winning.
- gweinberg 5y ago
  Is that 59 ply? 59 moves without a pawn move or capture means the game was drawn nine moves ago. Here's what's confusing me: if the chess engine has an initial numeric value for the current position, and the positions going out to the end of its search all have the same value or worse, shouldn't it be pretty trivial to conclude it is not making progress?
  - shric 5y ago
    Yes, I guess Stockfish doesn't bother to consider a draw on the 50 move rule. It does, iirc, go for a draw if it has no better via threefold repetition.

sillysaurusx 5y ago

When OpenAI trained their Dota model to beat people 1v1 mid SF, they had a contest at The International (the big yearly dota tournament, RIP) like "can you beat OpenAI's new bot?" and had a big bucket of prizes for anyone that could. By the end of the night, the bucket was empty. People learned to cheese the bot by running "out of bounds", so to speak -- normally in a 1v1, you're supposed to stay close to your opponent, since they'll be getting stronger if you leave. But the bot didn't know how to deal with it when you snuck behind it (normally an insane maneuver, almost guaranteed to cost you the game in normal play) and prevented his army (called "creeps") from running to the middle lane. (All you have to do is go wave your hand at them, and they mindlessly chase you.) People did that over and over, and your own army would eventually overwhelm the bot and win. :)

inostia 5y ago
That's a great story! It definitely speaks to human beings' knack for finding edge cases. Do you know what year that was? I'm wondering if AI has advanced and patched those weaknesses.
- sillysaurusx 5y ago
  I am of the surprisingly-controversial opinion that ML’s future is going to be as a human empowerment tool — a bicycle for the mind, yet so much more. Listen to how cool my gamedev music sounds: https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... I feel comfortable saying that it’s awesome and being immodest, because I didn’t write it! None of those notes are mine. “Crossing the Channel” has one of the strangest tempos I’ve heard, yet it sounds so cool because I think the model made a mistake early on, and decided it wasn’t a mistake — instead, it came up with the most likely not-mistake it could think of, and it happened to sound great. But I “wrote” all ~20 songs on that list in one night. Took about six hours, aka a standard workday. I don’t think it’s production-grade music - far from it - but it also wasn’t a production grade model. (On the other hand, it was trained by gwern, so I’m skeptical the production models of the future will capture all the little magic that he seems to imbue into his.) I chose the instruments, I decided what sounded good, but I didn’t write a single note. It felt a lot like listening to someone jam out, and asking them to play different things. I was just the fella who was there to listen. That’s where ML is going to shine. The future is going to be so exciting with the things you’ll be able to do. But not in the current direction, I think. Currently, the goal is to factor the human out of the equation entirely. So the answer to your question is: even if the AI has “advanced”, its only because they added this case to the training set. If a human was at the helm, they would have annihilated you, because they’d see what you were doing and pilot the AI over to you. I am skeptical this case is solvable, in a fully automatic way. IMO human empowerment will happen by arming actual humans with these models. (Depressingly, “arming” might have a few meanings depending on how AI war turns out, but that’s a different conversation.)

rudi-c 5y ago

There's a similar situation with Go where some positions utterly confuse bots trained on playing mostly normal games. There was this interesting research blog post on training a bot specifically to become good at solving one of these weird problems (Igo Hatsuyoron 120, the "hardest go problem ever") https://blog.janestreet.com/deep-learning-the-hardest-go-problem-in-the-world/ https://blog.janestreet.com/deep-learning-the-hardest-go-pro...

kadoban 5y ago
It is worth noting that that problem is only understood by humans (as well as it is) after _centuries_ of study by many professional players. My understanding is that even very strong human players must study that problem for days/months/years to really understand it well. So katago being unable to handle it without special training doesn't seem _quite_ as blind of a blindspot as the chess examples from the article seem to be (I suck at chess and I was able to understand one or two). I'm not trying to undermine you mentioning this, in case it comes off like that, on the contrary I think the comparison is quite interesting. I'm curious if this is just a difference in go vs chess, or in the relative abilities of specific kinds of AIs to handle these, or maybe just differences in human ability to craft and/or understand different problem difficulties between the games.

unnouinceput 5y ago

Quote: "Since IBM's Deep Blue defeated World Chess Champion Garry Kasparov in their 1997 match..." Deep Blue lost in 1996. Its upgrade, called Deeper Blue is the one that won the famous match in 1997. Please SamCopland, do your homework.

shric 5y ago
Do you have a source? Deeper Blue was only ever an unofficial nickname. The upgraded Deep Blue was still Deep Blue.
- unnouinceput 5y ago
  Per Wikipedia "heavy upgrade". Same like google of today is the same name as the one launched at beginning of 00's. Do you really think that from technological perspective is the same search engine or a heavy upgrade? The only thing Deep Blue and Deeper Blue have in common is that they were both put in the same case and created by IBM, everything else is different

johnklos 5y ago

Nice try, chess.com. I'm not going to let you use all those cycles stolen by your site to win chess games :P

vmception 5y ago

Yeah I would have thought these engines to be alot smarter then this, especially the “AI” ones

microtherion 5y ago

The Hasek study in #2 is bugging me: Why can't Black simply play 1. ... Rh8 ? It looks to me like this would gain the one crucial tempo to win the game. None of the discussion of the study I've found seems to consider this move (And the online analysis engines are useless, as they don't properly understand that the original line is a draw, as the article notes). ETA: Corrected typo, I originally wrote 1. ... Re8, which does not accomplish anything.

trykondev 5y ago
I think white just continues moving the king toward the center -- if black plays waiting moves, white can bring the king to c1-d1-e1 and then shuffle the king between e1 and f1. Eventually white will sacrifice the rook in the same pattern as the main line, and then play Ke2 -- there's no "wrong" square for the white king to be stuck on such that Ke2 would be unreachable, so black doesn't gain anything from the extra waiting moves.
- microtherion 5y ago
  Ah, sorry, I had a typo, I meant 1. ... Rh8, after which the rook sacrifice does not quite work the same way anymore. I agree that 1. ... Re8 does not help.
  - trykondev 5y ago
    Ah, understood -- I spent some time trying to figure it out after seeing your clarification. You are right that if you try the same rook sac technique, white won't be able to get the white king over in time to defend g2. It turns out there is another very impressive move white needs to spot: after 1...Rh8, white needs to find 2. Rf8! and now black has to either trade into the drawn king and pawn ending, or take the rook and give back the tempo that's just been saved. Now the white king is in time again. Quite a move, that should be listed as a sideline in the puzzle for sure!
    - microtherion 5y ago

deleted 5y ago

[deleted]

Andrex 5y ago

https://www.youtube.com/watch?v=Z4F7mUUjt_c https://www.youtube.com/watch?v=Z4F7mUUjt_c Maybe a more realistic scene than the fanbase gives it credit for. :)

aidenn0 5y ago
If it weren't a well-studied gambit perhaps...

zone411 5y ago

So I actually checked the problems listed. In the "IQ Test #52" position (FEN: 8/1p1q1k2/1Pp5/p1Pp4/P2Pp1p1/4PpPp/1N3P1P/3B2K1 w - - 0 1) listed in #1 both LC0 and Stockfish play the correct line on my computer in seconds. Both of them also play the right move Ba4+ in the second position of #1 "William Rudolph vs." (FEN: 8/1p1q1k2/1Pp5/p1Pp4/P2Pp1p1/4PpPp/1N3P1P/3B2K1 w - - 0 1) but they take quite a bit longer. Stockfish variants get it quicker. Stockfish solves "Hasek vs." (FEN: r7/7k/5R2/p3p3/Pp1pPp2/1PpP1Pp1/K1P3P1/8 w - - 0 1) listed in #2 quickly. Both Stockfish and LC0 solve "Lazard=F vs." (FEN: q7/8/2p5/B2p2pp/5pp1/2N3k1/6P1/7K w - - 0 1) quickly. Stockfish gets Bh3 from #3 "Veselin Topalov (?) vs. Alexey Shirov" (FEN: 8/8/4kpp1/3p1b2/p6P/2B5/6P1/6K1 b - - 2 47) in seconds (7-piece end game tablebases installed). The next position from #3 "Spassky, Boris V vs. Byrne, R." (FEN: 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P4P/PPB1K3/8 w - - 0 1) is easy for both Stockfish and LC0. They both get 50. c5!! right away. Stockfish also gets the last position from #3 "Stefan Brzozka vs. David Bronstein" in seconds (FEN: 1r6/4k3/r2p2p1/2pR1p1p/2P1pP1P/pPK1P1P1/P7/1B6 b - - 0 48) Rxb3+. Stockfish and LC0 see Kd1 in "Lamford=P vs." from #4 (FEN: 8/8/8/1k3p2/p1p1pPp1/PpPpP1Pp/1P1P3P/QNK2NRR w - - 0 1) but believe Rg2 is also winning. Stockfish gets c8N from "Randviir=J vs." (FEN: 5nr1/2Pp2pk/3Pp1p1/4P1P1/6P1/5K2/8/7n w - - 0 1) in #4 in about 2 minutes on my computer. Stockfish gets Bf5 from "Simkhovich=F vs." (FEN: 8/8/2pk4/8/p1p3B1/PpP5/1P6/r1NK4 w - - 2 2) in #5 in seconds. LC0 also gets it. The next two positions are mentioned as easier for programs and they are: Qe3 from "Deep Blue vs. Garry Kasparov" in #6 (FEN: 1r6/5kp1/RqQb1p1p/1p1PpP2/1Pp1B3/2P4P/6P1/5K2 b - - 14 45) is very easy for both Stockfish and LC0. Both also get "Vladimir Kramnik vs. Peter Leko" (FEN: 6k1/5p1p/P1pb1nq1/6p1/3P4/1BP2PP1/1P1Nb2P/R1B3K1 b - - 0 25) in #6 quickly. "Matous=M vs." (FEN: n2Bqk2/5p1p/Q4KP1/p7/8/8/8/8 w - - 0 1) is indeed harder for Stockfish and LC0 than expected. I've confirmed mate in 13 in my mate solver. In "Nigel Short vs. Vladimir Kramnik" (FEN: r3r1k1/1bp1Bppp/pb1p4/1p6/1P6/1BP2P2/P1P2PKP/R3R3 b - - 6 19) from #9, the engines like ...a5 more than ...c6. Hard to say that this move doesn't also win without more analysis. Stockfish wants to play c6! and b4! from "Marwitz=J vs." from #10 (FEN: 2K3k1/1p6/R3p1p1/1rB1P1P1/8/8/1Pb5/8 w - - 0 1) right from the start. LC0 takes longer but gets it as well. In "Anish Giri vs. Maxim Rodshtein" (FEN: 8/5pkp/3p1np1/Rpr5/8/6P1/PB3PKP/8 w - - 6 34) from #10 both Stockfish and LC0 like 34. h4 over 34. a4. More analysis would be needed to see if h4 is not also winning. Last position "IQ Test #16" (FEN: 5k2/4bp2/2B3p1/1P4p1/3R4/3P2PP/2r2PK1/8 b - - 0 1) takes around 10 seconds for Stockfish. In summary, chess engine might not really "understand" these positions but they solve them pretty well.

mrslave 5y ago

Agadmator covered Kramnik v Leko 2002 (in 2018) titled "Invisible to Engines | One Of The Greatest Moves Ever Played" [0] which is worth a watch. [0] https://www.youtube.com/watch?v=yGnpewUKP88 https://www.youtube.com/watch?v=yGnpewUKP88

ngcc_hk 5y ago

Politics are about making up rules that others have to follow. May be AI still not good at playing this level of politics.