4 ms·

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes?

• onos 4y ago
  Apparently not, at least at this level.
• fxtentacle 4y ago
  For Bomberland, we were quite surprised how strongly we could compress and quantize the current game state and still get useful movement predictions. I wouldn't be surprised if the relevant state in a typical beginner's chess game also excluded many units in the sense that yes, you could move them, but a beginner is going to just ignore them in any case.
• fsckboy 4y ago
  >playing a move from a different game in a new game is nonsensical GP did say "sequence of moves", and if it matches what it has seen from the first move on, including the opponent, it will be in a valid "sequence of moves". then, even midgame or endgame, if a sequence is played on one side of the board, even though the other side of the board may be different, the sequence has a great chance of being good (not always of course, but a 1400 rating is solid (you know the rules and some moves) but not amazing
• melenaboija 4y ago
  If it isn not memorizing, how do you think is doing it?
  • LeanderK 4y ago
    by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.
    • mrguyorama 4y ago
      It cannot be learning the general rules because it occasionally tries to invent pieces out of whole cloth.
      • LeanderK 4y ago
        that would be the optimal rule :) We usually optimise to improve and learn better and better rules, hopefully approximating the optimal rule after some iterations. There's still a gap here, but hopefully it can be closed by improving the models, training algorithms etc.
      • meh8881 4y ago
        Try asking it if a previous move was legal. Or why it decided to do the move. The problem is not a failure to understand the rules. It is just not very good at maintaining the state.
    • deleted 4y ago
      [deleted]
    • deleted 4y ago
      [deleted]
• qwytw 4y ago
  Is it though? I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..) I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.
  • medvezhenok 4y ago
    You could certainly test this by making completely random moves and seeing whether it's more likely to make illegal moves in those positions. Though I think you're overestimating how many positions have occured. Frequently, by move 20-25 you have a unique position that's never been played before (unless you're playing a well known main line or something)
  • sebzim4500 4y ago
    >I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..) Very low. On lichess when you analyse your games you can see which positions have been reached before, and you almost always diverge in the opening. The lichess db has orders of magnitude more games of chess than the chatGPT training data does, so there is absolutely no way that chatGPT could reach 1400 purely based off positions in its training data.
  • jointpdf 4y ago
    You can click through the Lichess opening database (click the book icon, and then the Lichess tab) to get an idea: https://lichess.org/analysis https://lichess.org/analysis But the answer is insanely unlikely, past a certain number of moves. The combinatorial explosion is inescapable. Even grandmaster games are often novelties in <10 moves. So, it has a to have some kind of internal representation of board state and what makes a reasonable move and such that enables it to generalize (choosing random legal moves is almost unbelievably bad, so it’s not doing that). I also doubt that it has been trained on the full (massive) database of Lichess games, but that would be an interesting experiment: https://database.lichess.org/ https://database.lichess.org/
• GauntletWizard 4y ago
  There's a pretty finite set of tokens that represent valid chess moves, and a more than countable set of possible opening moves, with a very distinct probability of moves; Nobody opens pg3, for example. Classical Markov chains played chess at some rate of success. ChatGPT is probably a lot better but not fundamentally different - It's predicting which moves to play based on sets of past games, not by memorizing it but by memoizing it.
• mtlmtlmtlmtl 4y ago
  1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not. These series of moves are referred to as opening theory and are described at copious length in tons of books. This is because while the raw number of possible paths to take is immense, the number of reasonable paths for 2 players of a given strength gets smaller and smaller. If I went over the 300 or so classical tournament games I've played I would ballmark that maybe just one or two would deviate from all known theory in the first 10 moves. So the criticism is valid in my view. The existence of copious chess literature can't simply be ignored here.
  • sebzim4500 4y ago
    Yeah but now explain how it played a 61 move game. EDIT: I checked and it left the lichess database after 9 moves. The lichess db has probably 5 orders of magnitude more chess games in it than chatGPT has in its training data.
    • mtlmtlmtlmtl 4y ago
      That's not the point. The point is if you truly want to test its strength, you'll have to control for these things. Maybe do things like invent a new form of notation and/or deliberately go into uncharted territory. Maybe start with a non-standard starting position even. Or play chess960 against it. In theory if I was playing a 1200 player I would almost always win, but let's say they have some extremely devious preparation that I fell into due to nonchalance and by the time we're both out of book I'm down a queen. It might not matter that I'm 600 points stronger at that point. If they don't make a sufficient amount of errors in return I will lose anyway.
      • sebzim4500 4y ago
        ChatGPT would probably play worse under those conditions, but then humans also get worse. ACPL is way higher at top level 960 events than at normal tournaments, for example
• btilly 4y ago
  Sorry, but not every game is unique. The following game has been played millions of times. 1. e4 e5 2. Bc4 Bc5 3. Qh5? Nf6?? 4. Qxf7++ The game Go has a claim to every game being unique. But not chess. And particularly not if both players follow a standard opening which there is a lot of theory about. Opening books often have lines 20+ moves deep that have been played many times. And grandmasters will play into these lines in tournament games so that they can reveal a novel idea that they came up with even farther in than that.
  • johnfn 4y ago
    You seem to be refuting a specific point of my argument which has little bearing on the overall point I was making. All games were provided in the article. None of them were 4 move checkmates; nearly every one is longer than 20 moves and some are 40 or longer. There is simply no possible way that ChatGPT is regurgitating the exact same 40-move-long game it's seen before. You can check a chess database if you'd like; virtually all games longer than 20 moves are unique.
    • mtlmtlmtlmtl 4y ago
      But this doesn't disprove that: 1. It definitely regurgitates opening theory, much more than can reasonably be calculated at its strength. 2. It might be regurgitating tactical sequences that appear in a lot of positions but remain identical in algebraic notation. Famous example: 1. Nxf7+ Kg8 2. Nh6++ Kh8 3. Qg8+ Rxg8 4. Nf7# This smothered mate can occur in a huge variety of different positions. There's some qualitative evidence for this in the games. In one of the games it has a bishop on f6 as white. It plays Qxh6?? Kxh6 and then resigns due to illegal move. I'd bet good money that illegal move was Rhx# where x is 1-4. So it seems like in some these positions it's filling in a tactical sequence that often occurs in the vicinity of recent moves, even when it's illegal or doesn't work tactically.