12 ms·

This really misses the key drama of what happened in the last game. Ding had a perfectly safe position where he could try to squeeze Gukesh pretty much endless

• tomatovole 2y ago
  Is there a metric I can look at in engine evaluations to determine when a situation is "risky" for white or black (e.g., the situation above) even if it looks equal with perfect play? I've always been interested in understanding situations where this is the case (and the opposite, where the engine favours one side but it seems to require a long, hard-to-find sequence of moves. Playing out the top lines helps if equality requires perfect play from one side.
  • Leary 2y ago
    https://live.lczero.org/ https://live.lczero.org/
    • fernandopj 2y ago
      This is great, but I think that % is about the "correctness" of the move, not how likely it is to be played next.
      • RUnconcerned 2y ago
        I think that's not quite the point. Leela has an advantage over AB chess engines, where it has multi-PV for "free", meaning it will evaluate multiple lines by default at no cost to performance (traditional engines, like Stockfish, will lose elo with multi-PV). This allows us to know at a glance if a position is "draw/win with perfect play" or if there is margin for error. If Leela shows multiple moves where one side maintains a winning advantage/losing disadvantage/equality, we can use that as a computer-based heuristic to know if a position is "easy" to play or not.
        • hilux 2y ago
          Yes and no – the number of playable lines does not necessarily tell us how "obvious" those lines are to find for a human. To give a trivial example, if I take your queen, then recapturing my queen is almost always the single playable move. But it's also a line that you will easily find! Conversely, in a complex tactical position, (even) multiple saving moves could all be very tricky for a human to calculate.
          • amohn9 2y ago
            I wonder if there’s a combined metric that could be calculated. Depth of the line certainly would be impactful. A line that only works if you do 5 only moves is harder to find than a single move line. “Quiet” moves are probably harder to find than captures or direct attacks. Backwards moves are famously tricky to spot. Etc
          • Scarblac 2y ago
            And also, humans vary wildly in their thinking and what's "obvious" to them. I'm about 1950 and am good in openings and tactics (but not tactics for the opponent). Others around the same rating are much worse than that but they understand positional play much better - how to use weak squares, which pieces to exchange and so on. To me that's a kind of magic.
  • nilslindemann 2y ago
    The metric is to play the position against Stockfish. If you draw it again and again, it is trivial, otherwise, not so simple :-)
  • jawarner 2y ago
    You can measure the sharpness of the position, as in this paper section 2.3 "Complexity of a position". They find their metric correlates with human performance. https://en.chessbase.com/news/2006/world_champions2006.pdf https://en.chessbase.com/news/2006/world_champions2006.pdf
    • jquery 2y ago
      Nice paper. I’d like if someone re-ran the numbers using modern chess engines… the engine they used is exceedingly weak by modern standards.
      • beyondCritics 2y ago
        Actually is is so weak, that it would be stomped out 1000:0 by modern engines. I like the methodology too, but the conclusions are not defendable.
    • somenameforme 2y ago
      I think this is something a bit different. That sort of assessment is going to find humans perform poorly in extremely sharp positions with lots of complicated lines that are difficult to evaluate. And that is certainly true. A tactical position that a computer can 'solve' in a few seconds can easily be missed by even very strong humans. But the position Ding was in was neither sharp nor complex. A good analog to the position there is the rook + bishop v rook endgame. With perfect play that is, in most cases, a draw - and there are even formalized drawing techniques in any endgame text. But in practice it's really quite difficult, to the point that even grandmasters regularly lose it. In those positions, on most of every move - any move is a draw. But the side with the bishop does have ways to inch up the pressure, and so the difficulty is making sure you recognize when you finally enter one of those moves where you actually need to deal with a concrete threat. The position Ding forced was very similar. Most of every move, on every move, led to a draw - until it didn't. Gukesh had all sorts of ways to try to prod at Ding's position and make progress - prodding Ding's bishop, penetrating with his king, maneuvering his bishop to a stronger diagonal, cutting off Ding's king, and of course eventually pushing one of the pawns. He was going to be able to play for hours just constantly prodding where Ding would have stay 100% alert to when a critical threat emerges. And this is all why Ding lost. His final mistake looks (and was) elementary, and he noticed it immediately after moving - but the reason he made that mistake is that he was thinking about how to parry the other countless dangerous threats, and he simply missed one. This is why most of everybody was shocked about Ding going for this endgame. It's just so dangerous in practical play, even if the computer can easily show you a zillion ways to draw it.
  • esfandia 2y ago
    Maybe the difference between the eval of the best move vs the next one(s)? An "only move" situation would be more risky than when you have a choice between many good moves.
    • fernandopj 2y ago
      That's it exactly. Engines will often show you at least 3 lines each with their valuation, and you can check the difficulty often just from that delta from 1st to 2nd best move. With some practical chess experience you can also "feel" how natural or exoteric the best move is. In the WCC match between Caruana and Carlsen, they were at one difficult endgame where Carlsen (the champion) moved and engines calculated it was a "blunder" because there was a theoretical checkmate in like 36(!) moves, but no commentator took it seriously as there was "no way" a human would be able to spot the chance and calculate it correctly under the clock.
    • kllrnohj 2y ago
      Not necessarily. If that "only move" is obvious, then it's not really risky. Like if a queen trade is offered and the opponent accepts, then typically the "only move" that doesn't massively lose is to capture back. But that's extremely obvious, and doesn't represent a sharp or complex position.
  • EGreg 2y ago
    Yes, it’s called Monte Carlo Tree Search (MCTS used by AlphaZero) instead of AlphaBeta search (which is what classical chess engines used)
    • elcomet 2y ago
      Those are tree search techniques, they are not metrics to assess the "human" complexity of a line. They could be used for this purpose but out of the box they just give you winning probability
      • RUnconcerned 2y ago
        If multiple lines have equal-ish winning probability, rather than a single line, then you can sort of translate it to "human" complexity.
  • hilux 2y ago
    Not really – that's the point, engines, for all their awesomeness, just do not know how to assess the likelihood of "human" mistakes.
  • qq66 2y ago
    Making a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.
    • dorgo 2y ago
      Take the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.
      • JonathanMerklin 2y ago
        You've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).
        • dmonitor 2y ago
          Yeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.
      • coffeeaddict1 2y ago
        This won't work. With that strategy, you can make a computer make play like a 1300 player, but not a 1300 human player.
      • qq66 2y ago
        That's kind of what they do for "training" bots and it produces something which plays NOTHING like a 1300-rated human.
      • oconnor663 2y ago
        I assume you could just give the computer a large set of 1300 rated games and train it to predict moves from that set :)
  • scott_w 2y ago
    Not really because it’s subjective to the level of player. What’s a blunder to a master player might only be an inaccuracy to a beginner. The same applies for higher levels of chess player. I’ve watched GothamChess say “I’ve no idea why <INSERT GM> made this move but it’s the only move,” then Hikaru Nakamura will rattle off a weird 8-move sequence to explain why it’s a major advantage despite no pieces being lost. Stockfish is a level above even Magnus if given enough depth.
    • School-Cotton 2y ago
      > Stockfish is a level above even Magnus if given enough depth. "a level" and "if given enough depth" are both underselling it. Stockfish running on a cheap phone with equal time for each side will beat Magnus 100 games in a row.
      • jquery 2y ago
        I believe it’s something like 500 elo points difference at this point between Magnus and Stockfish running on cheap hardware. Computers are so strong the only way to measure their strength is against other, weaker computers, and so on until you get to engines that are mere “grandmaster” strength.
        • scott_w 2y ago
          Bear in mind that, beyond the “top” elo ratings, that it’s purely an estimate of relative strength. The gap between a GM and me is far greater than the gap between a GM and Stockfish, even if the stated elo difference is the same. By this I mean, you can give me a winning position against Magnus and I’ll still lose. Give a winning position to Magnus vs Stockfish and he might draw or even win.
          • jquery 2y ago
            True, what is considered a “winning” position is different at different elo levels. The better someone is, the smaller their mistakes are relative to perfect play. I wish top players like Magnus would do more exhibition games against top computers. They don’t have to all start with equal material or an equal position.
  • paulddraper 2y ago
    You can evaluate on lower depth/time. But even that isn't a good proxy. Humans cannot out-FLOP a computer, so they need to use patterns (like an LLM). To get the human perspective, the engine would need to something similar.
    • lxgr 2y ago
      There are several neural network based engines these days, including one that does exclusively what you describe (i.e. "patterns only", no calculation at all), and one that's trained on human games. Even Stockfish uses a neural network these days by default for its positional evaluation, but it's relatively simple/lightweight in comparison to these, and it gains its strength from being used as part of deep search, rather than using a powerful/heavy neural network in a shallow tree search. [1] https://arxiv.org/html/2402.04494v1 https://arxiv.org/html/2402.04494v1 [2] https://www.maiachess.com/ https://www.maiachess.com/
      • paulddraper 2y ago
        Definitely. And Google's AlphaZero did it years ago. I don't think the patterns are very human, but they are very cool.
        • lxgr 2y ago
          Have you tried Maia? I haven't myself (there isn't one in my ballpark level yet), but supposedly it plays more human due to being trained mostly on human play, not engine evaluations or self-play.
          • paulddraper 2y ago
            I have not. Thank you.
  • rieska 2y ago
    Yes, the Leela team has worked on a term they call Contempt. (Negative contempt in this case would make the engine seek out less sharp play from whites perspective) In the first link the authour talks about using contempt to seek out/avoid sharp lines. lc0 and nibbler are free, so feel free to try it out if curious. https://github.com/LeelaChessZero/lc0/pull/1791#issuecomment-1486746586 https://github.com/LeelaChessZero/lc0/pull/1791#issuecomment... https://lczero.org/blog/2023/07/the-lc0-v0.30.0-wdl-rescale/contempt-implementation/ https://lczero.org/blog/2023/07/the-lc0-v0.30.0-wdl-rescale/...
• systemvoltage 2y ago
  Chess engines should come with another metric bar: "The twitchy-ness" of the position aka the gradient of primary eval metric as you pareto the possible moves from best to worst. The stronger this gradient, the more risky it is to play, and more changes to make a mistake.
  • staunton 2y ago
    This ignores the question how hard it is for a human to find the best (or a "good enough") move. It's easy to find games with 10 "only move" 's in a row where even a beginner could easily have played all if them.
    • __s 2y ago
      Sure, but it's a start on adding nuance to eval beyond minmax
      • WJW 2y ago
        Is it? TBH it sounds like "climbing a tree is a start on getting to the moon beyond just jumping up and down". Yes, it does "more". But whether it will actually get you to the desired end state is highly dubious. Nobody knows if that will make chess bots more human-like, despite decades of research into the topic.
        • __s 2y ago
          This is about eval nuance, not how bots play Bots playing like humans is done by training them to play like humans: https://lichess.org/@/lichess/blog/introducing-maia-a-human-like-neural-network-chess-engine/X9PUixUA https://lichess.org/@/lichess/blog/introducing-maia-a-human-...
  • Sesse__ 2y ago
    This is not a new request; many people, including engine authors, have suggested it throughout the years. The problem is that it's seemingly very hard to reliably quantify something like this and propagate it throughout the game tree.
• xmprt 2y ago
  Not the mention the time trouble that Ding left himself in once again. This time Gukesh ended with almost a full hour over Ding. When you put yourself in a tough position, no matter how drawish it is in theory, you need to have enough time to figure out the ideas of the position and with only 10 minutes left and 30 seconds per move, you might slip up and make a quick move when you really needed to think harder.
• jeremyjh 2y ago
  Yes and I think losing in this way is the most fair result. Ding has gone for a draw in every game where the score was tied, even with white (the first game, which he won as black was just a gift from Gukesh). Today, once again with white he could have pressed the position and played for more. Instead he sacrificed a pawn to play for a draw, and had the more difficult game to play even if it was always 0s. If he'd tried to play for a win today, almost certainly it would have been a draw anyway. While I was really happy to see Ding's fighting spirit in this match, and to have recovered much of his former strength, I've been rooting for Gukesh since around the half-way point just because Ding has not been playing superior positions for a win. I just don't think thats how a champion plays, even if its a sound strategy to try to win in tie-breaks.
  • mtlmtlmtlmtl 2y ago
    I think this is an unfair characterisation of how Ding played. The issue wasn't lack of ambition, but lack of confidence leading to misevaluation. Judging from the long thinks and how he's played in recent years, it's clear he doesn't fully trust his calculation. But I think he deserves credit for his ambition, actually. If he really wanted to play for draws he wouldn't be playing the French or the English. He'd be aiming for e4 e5 with an early queen trade. He mostly chose interesting openings with a lot of fight in them, often got an advantage, and simply misplayed by underestimating his position. Classic sign of a player with confidence issues.
    • hmm37 2y ago
      His long thinks is thought to be due to the fact that he simply hasn't been preparing. He stated he prepared for about 3 weeks for the championship match which was considered insanely low amount of time. But Caruana has stated he would be amazed if he even preped that long based on his games, and he always just looked like he was winging it every game.
    • jeremyjh 2y ago
      I watched the press conferences, and I agree that misevaluation was a big part of the problem. But even in game 6 for example, there is no way he could think he was not better after black refused the queen trade, and he just kept pressing for that trade. So yes, call that lack of confidence - but its still not what I want to see in the world champion.
• hmm37 2y ago
  It's strange/crazy because Ding even purposefully even gave up his B pawn, just so he could exchange queens and be in a 3 and 2 pawn game with a bishop and rook still in the game. Gukesh just tried playing out the game to the last second making easy moves while Ding suffered.
• hyperliner 2y ago
  [dead]
• moi2388 2y ago
  But in the last game it wasn’t about engine evaluation. It was a move that within 2 moves led to a known lost endgame..