4 ms·

Making a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.

by qq66 2y ago

dorgo 2y ago
Take the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.
- JonathanMerklin 2y ago
  You've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).
  - dmonitor 2y ago
    Yeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.
- coffeeaddict1 2y ago
  This won't work. With that strategy, you can make a computer make play like a 1300 player, but not a 1300 human player.
- qq66 2y ago
  That's kind of what they do for "training" bots and it produces something which plays NOTHING like a 1300-rated human.
- oconnor663 2y ago
  I assume you could just give the computer a large set of 1300 rated games and train it to predict moves from that set :)
  - Spivak 2y ago
    I think there's a real difference between "a computer"— in this context meaning an algorithm written by a human, possibly calibrated with a small number of parameters but not trained in any meaningful sense, and a "chess model" which works as you describe. I think the chess model would be successful at producing the desired outcome but it's not as interesting. There's something to be said for being able to write down in precise terms how to play imperfectly in a manner that feels like a single cohesive intelligence strategizing against you.
- WJW 2y ago
  Even 1300s sometimes make the best move. Sometimes the best move is really easy to see or even mandatory, like if you are in check and MUST take that checking piece. Sometimes the best move is only obvious if you can look 20 moves ahead. Sometimes the best move is only obvious if you can look 5 moves ahead, but the line is so forcing that even 1300s can look that far ahead. Despite decades of research, nobody has found a good way to make computers play like humans.
  - throw310822 2y ago
    Then I can't refrain from asking: and what's the style of LLMs? For example the ChatGPT which is apparently rated around 1800? That should be completely different from that of a classic chess engine.
    - ludston 2y ago
      ChatGPT will hallucinate and make impossible/invalid moves frequently, so I don't see how it could have a chess rating
      - throw310822 2y ago
        That's not the case. Depending on the version, (Chat)GPT seems to be able to play between ~1400 and ~1800 elo, very rarely making invalid moves.
    - jquery 2y ago
      I can’t speak for ChatGPT, but your intuition is correct that LLMs tend to play more like “humans” than Stockfish or other semi-brute force approaches.
    - WJW 2y ago
      LLMs can be trained on chess games, but the tree of possible board states branches so fast that for any given position there is simply very little training data available. Even the billions of games played on chess.com and lichess are only a drop in the bucket compared to how many possible board states there are. This would have to be split further by rating range, so the amount of games for any given rating range would be even lower. This means that the LLM does not actually have a lot of training data available to learn how a 1300 would play, and subsequently does a poor job at imitating it. There is a bunch of papers available online if you want more info.
SpaceManNabs 2y ago
This is really interesting because i ran into a pokemon bot the other day were its training led to calibration of 50% winrste at all levels of play on Pokémon showdown. It was a complete accident.
- Someone 2y ago
  But that doesn’t imply that that bot played like an average human. Making a computer have a 50% score against a 1300-rated human is way easier than making it play like a 1300-rated human. For the former, you can take a top-of-the-line program and have it flip a coin in every game whether to make a random move every move or not.
- qq66 2y ago
  Playing a chess bot that works this way feels like playing a Magnus Carlsen who's trying to let you win.
- School-Cotton 2y ago
  It's not hard to make a chess bot that plays at a 1300 strength, i.e. its rating would converge to 1300 if it were allowed to compete. But it will not play like a 1300-rated human. It would play like a superhuman genius on most moves and then make beginner-level blunders at random moments. Making one that realistically plays like a human is an unsolved problem.
  - darkerside 2y ago
    So like a 1500 rated human?
  - SpaceManNabs 2y ago
    ah that makes sense. thanks!
  - rieska 2y ago
    Of course, you are right. But (the linked site) at least has a bot that plays the opening like a human of chosen rating perfectly. It stops working after the opening-stage (since it just copies moves from humans in the lichess game database), but it is still very impressive. For later game stages, some other method would have to be used (unless we play multiple orders of magnintude more games on lichess). Now that i think about it, i remember the people in the alphago documentary talking about the bot giving its moves percentage scores in both how high winning % the move had and how high % chance that a human would have made the same move that it just played. I wonder why they never showed what a full game of the most human-like moves from alphago would look like. Maybe it actually worked, by feeding it all the pro games in existence, and training it to play the high human % instead of the higest win probability moves like they did in the end. https://www.chessassess.com/openings https://www.chessassess.com/openings
lxgr 2y ago
Definitely, but it seems like it's now possible: https://www.maiachess.com/ https://www.maiachess.com/

Making a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.

by qq66 2y ago

Making a computer play like a 1300-rated human is harder than making a computer beat Magnus Carlsen.

Take the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.

JonathanMerklin 2y ago
You've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).
- dmonitor 2y ago
  Yeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.
coffeeaddict1 2y ago
This won't work. With that strategy, you can make a computer make play like a 1300 player, but not a 1300 human player.
qq66 2y ago
That's kind of what they do for "training" bots and it produces something which plays NOTHING like a 1300-rated human.
oconnor663 2y ago
I assume you could just give the computer a large set of 1300 rated games and train it to predict moves from that set :)
- Spivak 2y ago
  I think there's a real difference between "a computer"— in this context meaning an algorithm written by a human, possibly calibrated with a small number of parameters but not trained in any meaningful sense, and a "chess model" which works as you describe. I think the chess model would be successful at producing the desired outcome but it's not as interesting. There's something to be said for being able to write down in precise terms how to play imperfectly in a manner that feels like a single cohesive intelligence strategizing against you.
WJW 2y ago
Even 1300s sometimes make the best move. Sometimes the best move is really easy to see or even mandatory, like if you are in check and MUST take that checking piece. Sometimes the best move is only obvious if you can look 20 moves ahead. Sometimes the best move is only obvious if you can look 5 moves ahead, but the line is so forcing that even 1300s can look that far ahead. Despite decades of research, nobody has found a good way to make computers play like humans.
- throw310822 2y ago
  Then I can't refrain from asking: and what's the style of LLMs? For example the ChatGPT which is apparently rated around 1800? That should be completely different from that of a classic chess engine.
  - ludston 2y ago
    ChatGPT will hallucinate and make impossible/invalid moves frequently, so I don't see how it could have a chess rating
    - throw310822 2y ago
      That's not the case. Depending on the version, (Chat)GPT seems to be able to play between ~1400 and ~1800 elo, very rarely making invalid moves.
  - jquery 2y ago
    I can’t speak for ChatGPT, but your intuition is correct that LLMs tend to play more like “humans” than Stockfish or other semi-brute force approaches.
  - WJW 2y ago
    LLMs can be trained on chess games, but the tree of possible board states branches so fast that for any given position there is simply very little training data available. Even the billions of games played on chess.com and lichess are only a drop in the bucket compared to how many possible board states there are. This would have to be split further by rating range, so the amount of games for any given rating range would be even lower. This means that the LLM does not actually have a lot of training data available to learn how a 1300 would play, and subsequently does a poor job at imitating it. There is a bunch of papers available online if you want more info.

SpaceManNabs 2y ago

This is really interesting because i ran into a pokemon bot the other day were its training led to calibration of 50% winrste at all levels of play on Pokémon showdown. It was a complete accident.

Someone 2y ago
But that doesn’t imply that that bot played like an average human. Making a computer have a 50% score against a 1300-rated human is way easier than making it play like a 1300-rated human. For the former, you can take a top-of-the-line program and have it flip a coin in every game whether to make a random move every move or not.
qq66 2y ago
Playing a chess bot that works this way feels like playing a Magnus Carlsen who's trying to let you win.
School-Cotton 2y ago
It's not hard to make a chess bot that plays at a 1300 strength, i.e. its rating would converge to 1300 if it were allowed to compete. But it will not play like a 1300-rated human. It would play like a superhuman genius on most moves and then make beginner-level blunders at random moments. Making one that realistically plays like a human is an unsolved problem.
- darkerside 2y ago
  So like a 1500 rated human?
- SpaceManNabs 2y ago
  ah that makes sense. thanks!
- rieska 2y ago
  Of course, you are right. But (the linked site) at least has a bot that plays the opening like a human of chosen rating perfectly. It stops working after the opening-stage (since it just copies moves from humans in the lichess game database), but it is still very impressive. For later game stages, some other method would have to be used (unless we play multiple orders of magnintude more games on lichess). Now that i think about it, i remember the people in the alphago documentary talking about the bot giving its moves percentage scores in both how high winning % the move had and how high % chance that a human would have made the same move that it just played. I wonder why they never showed what a full game of the most human-like moves from alphago would look like. Maybe it actually worked, by feeding it all the pro games in existence, and training it to play the high human % instead of the higest win probability moves like they did in the end. https://www.chessassess.com/openings https://www.chessassess.com/openings

lxgr 2y ago

Definitely, but it seems like it's now possible: https://www.maiachess.com/ https://www.maiachess.com/