4 ms·

1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long ti

1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not. These series of moves are referred to as opening theory and are described at copious length in tons of books. This is because while the raw number of possible paths to take is immense, the number of reasonable paths for 2 players of a given strength gets smaller and smaller. If I went over the 300 or so classical tournament games I've played I would ballmark that maybe just one or two would deviate from all known theory in the first 10 moves. So the criticism is valid in my view. The existence of copious chess literature can't simply be ignored here.

sebzim4500 4y ago
Yeah but now explain how it played a 61 move game. EDIT: I checked and it left the lichess database after 9 moves. The lichess db has probably 5 orders of magnitude more chess games in it than chatGPT has in its training data.
- mtlmtlmtlmtl 4y ago
  That's not the point. The point is if you truly want to test its strength, you'll have to control for these things. Maybe do things like invent a new form of notation and/or deliberately go into uncharted territory. Maybe start with a non-standard starting position even. Or play chess960 against it. In theory if I was playing a 1200 player I would almost always win, but let's say they have some extremely devious preparation that I fell into due to nonchalance and by the time we're both out of book I'm down a queen. It might not matter that I'm 600 points stronger at that point. If they don't make a sufficient amount of errors in return I will lose anyway.
  - sebzim4500 4y ago
    ChatGPT would probably play worse under those conditions, but then humans also get worse. ACPL is way higher at top level 960 events than at normal tournaments, for example
    - mtlmtlmtlmtl 4y ago
      This is true. But I think ChatGPT compared to a human player of equivalent strength(leaving the dubious number 1400 out for now) will probably have a lot more opening knowledge than the human. So it would be interesting to eliminate all opening knowledge and that way be able to qualitately get at which aspects of chess it's actually good at, which is sucks out, and how much of its strength can be attributed to opening knowledge. I'm still impressed by this btw. I did not expect this to be possible at all really. But being impressed is not an excuse to ignore methodological flaws. :)
- roflyear 4y ago
  It's suggesting a move it saw come after a move somewhere. Or it's creating a token it thinks comes next.
johnfn 4y ago
I'm very familiar with opening theory. Some of the games are 40 or 60 movies. This is not a regurgitation of book moves.
- mtlmtlmtlmtl 4y ago
  Why do people always have to interpret everything in absolute terms? It's clearly following some opening theory in all the games I've looked at so far. So yes, it is regurgitating opening moves. That's clearly not all it's doing, which is very impressive, but these are not mutually exclusive.
  - johnfn 4y ago
    I am responding to OP, who said "Most likely it has seen a similar sequence of moves in its training set." From this, I take it that the question is if ChatGPT is repeating existing games, or not. All you need is a single game where it's not repeating a single game to prove it definitively. You can hardly play 60 moves without an error by accident. I believe you're responding to a different question, something like "does ChatGPT fully understand the game of chess".
    - mtlmtlmtlmtl 4y ago
      The OP was too unsophisticated in their analysis(as is TFA), no doubt. But I'm not too interested in what OP said or who was wrong or not, and rather more interested in finding what's right. As someone very clever once said, welcome to the end of the thought process. We've established that: 1. It doesn't repeat entire games when the games go long enough 2. It does repeat a lot of opening theory 3. It seems to repeat common, partially position independent tactical sequences even when they're illegal or don't work tactically.

sebzim4500 4y ago

Yeah but now explain how it played a 61 move game. EDIT: I checked and it left the lichess database after 9 moves. The lichess db has probably 5 orders of magnitude more chess games in it than chatGPT has in its training data.

mtlmtlmtlmtl 4y ago
That's not the point. The point is if you truly want to test its strength, you'll have to control for these things. Maybe do things like invent a new form of notation and/or deliberately go into uncharted territory. Maybe start with a non-standard starting position even. Or play chess960 against it. In theory if I was playing a 1200 player I would almost always win, but let's say they have some extremely devious preparation that I fell into due to nonchalance and by the time we're both out of book I'm down a queen. It might not matter that I'm 600 points stronger at that point. If they don't make a sufficient amount of errors in return I will lose anyway.
- sebzim4500 4y ago
  ChatGPT would probably play worse under those conditions, but then humans also get worse. ACPL is way higher at top level 960 events than at normal tournaments, for example
  - mtlmtlmtlmtl 4y ago
    This is true. But I think ChatGPT compared to a human player of equivalent strength(leaving the dubious number 1400 out for now) will probably have a lot more opening knowledge than the human. So it would be interesting to eliminate all opening knowledge and that way be able to qualitately get at which aspects of chess it's actually good at, which is sucks out, and how much of its strength can be attributed to opening knowledge. I'm still impressed by this btw. I did not expect this to be possible at all really. But being impressed is not an excuse to ignore methodological flaws. :)
roflyear 4y ago
It's suggesting a move it saw come after a move somewhere. Or it's creating a token it thinks comes next.

johnfn 4y ago

I'm very familiar with opening theory. Some of the games are 40 or 60 movies. This is not a regurgitation of book moves.

mtlmtlmtlmtl 4y ago
Why do people always have to interpret everything in absolute terms? It's clearly following some opening theory in all the games I've looked at so far. So yes, it is regurgitating opening moves. That's clearly not all it's doing, which is very impressive, but these are not mutually exclusive.
- johnfn 4y ago
  I am responding to OP, who said "Most likely it has seen a similar sequence of moves in its training set." From this, I take it that the question is if ChatGPT is repeating existing games, or not. All you need is a single game where it's not repeating a single game to prove it definitively. You can hardly play 60 moves without an error by accident. I believe you're responding to a different question, something like "does ChatGPT fully understand the game of chess".
  - mtlmtlmtlmtl 4y ago
    The OP was too unsophisticated in their analysis(as is TFA), no doubt. But I'm not too interested in what OP said or who was wrong or not, and rather more interested in finding what's right. As someone very clever once said, welcome to the end of the thought process. We've established that: 1. It doesn't repeat entire games when the games go long enough 2. It does repeat a lot of opening theory 3. It seems to repeat common, partially position independent tactical sequences even when they're illegal or don't work tactically.