4 ms·

The author is looking for positions which are difficult for low rated players and easier for high rated players. Poor man’s version of this which requires no t

The author is looking for positions which are difficult for low rated players and easier for high rated players. Poor man’s version of this which requires no training would be to evaluate positions at low depth and high depth and select positions where the best move switches. Training neural nets to model behavior at different levels is also possible but high rated players are inherently more difficult to model.

nilslindemann 1y ago
Seems like an excellent definition, mainly because I have no idea how to measure it otherwise. "Complexity is the time needed to solve a problem", why not.
- Taek 1y ago
  Well, there are a lot of shortcuts built into engines like stockfish that don't perfectly mirror how beginners vs advanced players think. For example, certain types of piece forks are easy for newer players to spot (knight forking a king+rook on the back rank) and others are harder (bishop forking a king and a pawn in the late game when nobody is on the back rank), but stockfish is going to find both just as easily. So if you want to get deep into complexity you do need a model of different players and what types of advantages are easy to see for them.
  - nilslindemann 1y ago
    It could be done with e.g. the Maia engine, which was trained on Lichess games and plays like humans on different skill levels – https://www.maiachess.com/ https://www.maiachess.com/ – but I do not know if it supports different ponder depths. Edit: it seems it does not – https://github.com/CSSLab/maia-chess#:~:text=a%20nodes%20limit%20of%201%20is%20what%20we%20use https://github.com/CSSLab/maia-chess#:~:text=a%20nodes%20lim...
chongli 1y ago
evaluate positions at low depth and high depth and select positions where the best move switches. Won’t that bias in favour of common sacrifices in the endgame? The issue with using depth is that humans don’t struggle with depth uniformly. Humans can calculate to a lot more depth with a sequence of obvious moves (recaptures, pawn races) than they can in more complex situations (closed positions in the midgame).
- janalsncm 1y ago
  Yeah, this is called the horizon effect: if I take your knight with my queen and stop calculating I’m up a knight, when in reality the queen will be taken on the next turn.
  - lazyant 1y ago
    One chess engine I saw long ago used a small depth (maybe even 1!) but with the exception of consecutive captures to avoid the biggest case for this issue (pawn promotion as well iirc)
- nilslindemann 1y ago
  Seems to me that if it is common, it is better known, so the AI knows it better, so it is less komplex.
adgjlsfhk1 1y ago
> high rated players are inherently more difficult to model yes and no. there's a bit of a bathtub curve here where below ~1000 elo lower ratings are harder to predict because their moves are closer to random.
- kelipso 1y ago
  I don't even think "high rated players are inherently more difficult to model" is correct, and the opposite is more likely to be correct. Top player, more than 2700 players, tend to play the engine moves the majority of the time, while lower rated players tend to stray from the engine lines.
  - adgjlsfhk1 1y ago
    top players are hard to model just because there aren't many of them. 1200-2200 has tons of players and they all make various degrees of sensible moves most of the time, so you can plausibly train a NN to sample from those distributions. for 2500+ you have far fewer players so the likely moves depend a lot more on the styles and blindspots of specific players.
  - janalsncm 1y ago
    https://arxiv.org/pdf/2006.01855 https://arxiv.org/pdf/2006.01855 If you check figure 4, SF at depth 15 matches about 47% of the time which is lower than their NN. So quantitatively, SF isn’t amazing at move matching, and lower than the best NN at its relevant rating range (1900 -> 54%). Qualitatively, SF is pretty unsatisfying as a model of a pro human because a lot of its moves are counterintuitive.
codechicago277 1y ago
I had this idea of drilling games against an engine with a set depth evaluation, since beating a depth 1 engine should teach simpler concepts than level 4. I vibe coded this into a browser app, but the evaluation is slow around depth 5: https://camjohnson26.github.io/chess-trainer/ https://camjohnson26.github.io/chess-trainer/
- icelancer 1y ago
  This is a thing you can do with Stockfish + SCID, is it not?
  - nilslindemann 1y ago
    Yes, though in the latest versions it is just a drop-down where one can choose between a few fixed values, and one has to make the move for the engine. Cute chess is a pretty GUI where one can do this too by setting the ply parameter. Castling is done by dragging the king on the rook! https://github.com/cutechess/cutechess https://github.com/cutechess/cutechess
kadoban 1y ago
> Poor man’s version of this which requires no training would be to evaluate positions at low depth and high depth and select positions where the best move switches. I've looked at the ~same problem in go instead of chess quite a bit. It turns out that that strategy does remarkably poorly, mostly because weak players don't just have low-depth thinking, they also do random weird shit and miss entire patterns. Eg in go if you run a strong go engine at _zero_ thinking (just take the model's best guess first move), it's already damn close to pro level. Getting them to play anything like a beginner, or even intermediate player is functionally impossible. You _may_ be able to trick one into playing as weak as a beginner player if you really try, but it would feel _nothing_ like one. > Training neural nets to model behavior at different levels is also possible but high rated players are inherently more difficult to model. Maybe more difficult to model (not sure tbh, but granted for the moment), but it's _far_ easier to generate training data for strong players via reinforcement learning approaches.
- vintermann 1y ago
  In go, I remember that there were rare positions/puzzles where old (pre-mcts) algorithms did better than humans, and that was artificial positions where there were a lot of complex recaptures in the same area. Which is rare in go, but common in chess. I do think that positions where the best move at depth N is a terrible move at N+1 are hard, especially if it isn't just recaptures on the same square. Chess engines used to have special heuristics to avoid such "horizon effects", don't know if they still do.
- janalsncm 1y ago
  > Maybe more difficult to model (not sure tbh, but granted for the moment), Two reasons. One, strong players are performing tree search, so you need to model their tree search process rather than a simple depth zero prediction. And two, there are far fewer high rated players so there is far less high quality training data. A lot of high rated games are also bullet, which aren’t as useful. > but it's _far_ easier to generate training data for strong players via reinforcement learning approaches. Can you say more about this?

nilslindemann 1y ago

Seems like an excellent definition, mainly because I have no idea how to measure it otherwise. "Complexity is the time needed to solve a problem", why not.

Taek 1y ago
Well, there are a lot of shortcuts built into engines like stockfish that don't perfectly mirror how beginners vs advanced players think. For example, certain types of piece forks are easy for newer players to spot (knight forking a king+rook on the back rank) and others are harder (bishop forking a king and a pawn in the late game when nobody is on the back rank), but stockfish is going to find both just as easily. So if you want to get deep into complexity you do need a model of different players and what types of advantages are easy to see for them.
- nilslindemann 1y ago
  It could be done with e.g. the Maia engine, which was trained on Lichess games and plays like humans on different skill levels – https://www.maiachess.com/ https://www.maiachess.com/ – but I do not know if it supports different ponder depths. Edit: it seems it does not – https://github.com/CSSLab/maia-chess#:~:text=a%20nodes%20limit%20of%201%20is%20what%20we%20use https://github.com/CSSLab/maia-chess#:~:text=a%20nodes%20lim...

chongli 1y ago

evaluate positions at low depth and high depth and select positions where the best move switches. Won’t that bias in favour of common sacrifices in the endgame? The issue with using depth is that humans don’t struggle with depth uniformly. Humans can calculate to a lot more depth with a sequence of obvious moves (recaptures, pawn races) than they can in more complex situations (closed positions in the midgame).

janalsncm 1y ago
Yeah, this is called the horizon effect: if I take your knight with my queen and stop calculating I’m up a knight, when in reality the queen will be taken on the next turn.
- lazyant 1y ago
  One chess engine I saw long ago used a small depth (maybe even 1!) but with the exception of consecutive captures to avoid the biggest case for this issue (pawn promotion as well iirc)
nilslindemann 1y ago
Seems to me that if it is common, it is better known, so the AI knows it better, so it is less komplex.

adgjlsfhk1 1y ago

> high rated players are inherently more difficult to model yes and no. there's a bit of a bathtub curve here where below ~1000 elo lower ratings are harder to predict because their moves are closer to random.

kelipso 1y ago
I don't even think "high rated players are inherently more difficult to model" is correct, and the opposite is more likely to be correct. Top player, more than 2700 players, tend to play the engine moves the majority of the time, while lower rated players tend to stray from the engine lines.
- adgjlsfhk1 1y ago
  top players are hard to model just because there aren't many of them. 1200-2200 has tons of players and they all make various degrees of sensible moves most of the time, so you can plausibly train a NN to sample from those distributions. for 2500+ you have far fewer players so the likely moves depend a lot more on the styles and blindspots of specific players.
- janalsncm 1y ago
  https://arxiv.org/pdf/2006.01855 https://arxiv.org/pdf/2006.01855 If you check figure 4, SF at depth 15 matches about 47% of the time which is lower than their NN. So quantitatively, SF isn’t amazing at move matching, and lower than the best NN at its relevant rating range (1900 -> 54%). Qualitatively, SF is pretty unsatisfying as a model of a pro human because a lot of its moves are counterintuitive.

codechicago277 1y ago

I had this idea of drilling games against an engine with a set depth evaluation, since beating a depth 1 engine should teach simpler concepts than level 4. I vibe coded this into a browser app, but the evaluation is slow around depth 5: https://camjohnson26.github.io/chess-trainer/ https://camjohnson26.github.io/chess-trainer/

icelancer 1y ago
This is a thing you can do with Stockfish + SCID, is it not?
- nilslindemann 1y ago
  Yes, though in the latest versions it is just a drop-down where one can choose between a few fixed values, and one has to make the move for the engine. Cute chess is a pretty GUI where one can do this too by setting the ply parameter. Castling is done by dragging the king on the rook! https://github.com/cutechess/cutechess https://github.com/cutechess/cutechess

kadoban 1y ago

> Poor man’s version of this which requires no training would be to evaluate positions at low depth and high depth and select positions where the best move switches. I've looked at the ~same problem in go instead of chess quite a bit. It turns out that that strategy does remarkably poorly, mostly because weak players don't just have low-depth thinking, they also do random weird shit and miss entire patterns. Eg in go if you run a strong go engine at _zero_ thinking (just take the model's best guess first move), it's already damn close to pro level. Getting them to play anything like a beginner, or even intermediate player is functionally impossible. You _may_ be able to trick one into playing as weak as a beginner player if you really try, but it would feel _nothing_ like one. > Training neural nets to model behavior at different levels is also possible but high rated players are inherently more difficult to model. Maybe more difficult to model (not sure tbh, but granted for the moment), but it's _far_ easier to generate training data for strong players via reinforcement learning approaches.

vintermann 1y ago
In go, I remember that there were rare positions/puzzles where old (pre-mcts) algorithms did better than humans, and that was artificial positions where there were a lot of complex recaptures in the same area. Which is rare in go, but common in chess. I do think that positions where the best move at depth N is a terrible move at N+1 are hard, especially if it isn't just recaptures on the same square. Chess engines used to have special heuristics to avoid such "horizon effects", don't know if they still do.
janalsncm 1y ago
> Maybe more difficult to model (not sure tbh, but granted for the moment), Two reasons. One, strong players are performing tree search, so you need to model their tree search process rather than a simple depth zero prediction. And two, there are far fewer high rated players so there is far less high quality training data. A lot of high rated games are also bullet, which aren’t as useful. > but it's _far_ easier to generate training data for strong players via reinforcement learning approaches. Can you say more about this?