4 ms·

Centipawn loss (or simply the engine's evaluation of a position) doesn't take into account how realistically a human could hold a position. During yesterday's

• polytely 5y ago
  and exhaustion also starts to play a role after playing for like 7 hours, at some point mistakes will start to slip in.
• jeremyjh 5y ago
  You are definitely right about the evaluation; I switched between several streams and I don't think there was anyone saying that they didn't prefer white after around move 39, despite a 0.0 eval for a lot of those positions. But part of the reason the eval is misleading is because it might be reflecting a sequence of "only moves" - where only one move can hold that evaluation and it may be very hard to find some of those moves for black, while white has lots of good moves in each position. While that is a problem with human interpretation of eval, I do not see how it invalidates use of ACPL which is an average across the entire game.
• eckesicle 5y ago
  I was thinking a bit about this during the game too. Perhaps alongside centipawn loss (a measure of how many hundredths of a pawn a player loses by making the non-optimal move as determined by a chess AI engine) we could also measure the difficulty of any position. Stockfish (a popular chess engine) roughly works by constructing a tree of possible moves and evaluating the score according to some heuristic at its maximum depth. The best result at depth n (25 I believe) is considered the best move and incurs 0 centipawn loss. Perhaps we can define the difficulty of a position by the relative centipawn loss at each preceeding depth in the tree? The difficulty of a position is then determined by the depth at which the best move no longer changes.
  • ashtonbaker 5y ago
    This is an interesting thought! A couple of other scattered thoughts I had about this: - Engine evaluation of a leaf of the tree will always be different and more sophisticated than human heuristics. So there's a problem where a human can't be expected to follow down some lines. Of course, this is always changing, as humans seek to understand engine heuristics better. Carlsen's "blunder" at move 33 was a good example of this, from my memory. - Maybe there's a difficulty metric like "sharpness", some function of the number of moves which do not incur a significant centipawn loss. Toward the end of game 6, Carlsen faced a relatively low sharpness on his moves, whereas Nepomniachtchi faced a high sharpness, and despite the theoretical draw, this difference will prove to be decisive between humans. This seems like it could interact in interesting ways with your difficulty metric - for example, what does it mean if sharpness is only revealed at high depth? - It would be interesting to take the tree generated by stockfish, and weight the tree at each node by the probability that a human player would evaluate the position as winning. Then you could give a probability of ending up at each terminal position of the tree. Maybe some sort of deep learning model trained on players previous games? Time controls add such a confounding factor to this, but it would be so interesting to see "wild engine lines" highlighted in real-time.
    • ludamad 5y ago
      You could calculate difficulty by having some amount of dials on engine strength (lookahead, eval complexity) and fitting them to a function of approximate strength. You could then try to plot how the position "real outcome" changes. You could also try to use different engines or otherwise introduce variance as this sort of thing can be a tad deterministic for stats
• 4by4by4 5y ago
  In my opinion, neural network engines like Alpha Zero or Leela Zero do a much better job of assessing how difficult a position is to hold. They also report the evaluation in a completely different manner, win/draw/loss probability as opposed to centipawn loss. For example, in yesterday’s game Stockfish was often giving a drawn evaluation (0.00) where Leela Chess gave a win probability of 30%+. I was posting about this during the game. https://twitter.com/nik_king_/status/1466794534214504454?s=20 https://twitter.com/nik_king_/status/1466794534214504454?s=2...
  • LudwigNagasena 5y ago
    Stockfish also uses a neural network for evaluation.
    • 4by4by4 5y ago
      True, it’s works differently though. And Stockfish does not compute win/draw/loss probabilities as part of its eval. It converts cp to wdl using an exponential fit based stockfish vs stockfish games. So draw percentage in Leela is a lot more interesting and useful.
    • ummonk 5y ago
      Yes, but at its heart it's still classical alpha-beta search rather than monte carlo.
• nextaccountic 5y ago
  Why isn't the time left of each player an input to the evaluator It shouldn't assume that everyone has plenty of time!
  • 4by4by4 5y ago
    I’ve thought about this, too. Sadly, times only started being recorded in the last 20 years and last time I tried I couldn’t find a large dataset with the times included. Chess.com did do one study and found a large percentage of mistakes occurs in moves 36-40 because in some time controls additional time is added at move 40.
• Someone 5y ago
  > In the final stages the computer again evaluated the game as drawn, but presumed Nepo could defend perfectly for tens of moves without a single inaccuracy. I agree that it’s not very useful to compare with table bases, especially given the “30 seconds time added per move” regime this was played under by the time they reached the position. However, I don’t think the table bases even have enough information to indicate how close to losing a theoretically drawn position is. So, i don’t think this required perfect accuracy to defend against (defining ‘inaccuracy’ as any move for black that either makes it take longer to reach a draw or moves to a losing position. That, I think, is the most reasonable definition)
• oneoff786 5y ago
  This is merely the seaborn “darkgrid” style option. You need to set it explicitly if you want this effect.