5 ms·
It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't
by Jensson 4y ago
It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.
- SanderNL 4y agoAgain, not enough for 1400. 800 maybe.
- Jensson 4y agoWe have evidence right here that patterns of moves leads to 1400. To me this is evidence that 1400 level play is very rigid and mostly relies on defined openings and strategies, making it easy for an LLM to find similar games and win using standard continuations. Just like grammar the patterns are too hard for humans to see and encode, but LLMs can encode pretty complex patterns. Domains that are easy to encode as grammars will be really easy for LLMs to solve, and the further from a grammar the harder for it.
- SanderNL 4y agoAt what point can we just say that understanding “patterns of moves” is understanding chess? It seems you suggest there is more to it, but maybe I am mistaken.
- Jensson 4y agoAt least it should make valid moves, that is the minimum level required. It didn't reach that level here. If it never made illegal moves we could talk and see what it does, but until then we can be sure it didn't understand the rules.
- serverholic 4y agoI don’t understand why the threshold is “never”. Isn’t it entirely possible that the AI is learning a model of chess but this model is imperfect? What if AIs don’t fail the same way as humans?
- Jensson 4y ago> What if AIs don’t fail the same way as humans? But it is failing the same way as a human. Humans who remembers patterns and don't learn the logic makes these kind of errors in math or logic all the time. ChatGPT is much better than humans at pattern matching, you see it right here it can pattern match chess moves and win games! But its inability to apply logic to its output instead of just pattern matching is holding it back, as long as that isn't solved it wont be able to perform on the level of humans in many tasks. Chess might be easy enough to solve using just pattern matching and no logic that scaling it up will make it pretty good at chess, but many other topics wont be.
- meh8881 4y agoAsk it to evaluate moves and explain why they are or are not good
- lelandfe 4y agoPerhaps I don’t know what “easy” means in your comment, but the OP article says that they “almost always” played full, legal games with their prompt.
- Jensson 4y agoMaybe chess.com players play in a specific way, and there are lot of transcriptions of such games that these LLMs have ingested when they ingested the internet? I don't know why it worked in this specific case, but based on earlier examples it is more likely that these kind of games were more prevalent in its dataset it was trained on than it being able to play chess in general. It still wasn't perfect, so even these games weren't rigid enough for it to reliably perform valid moves.
- antibasilisk 4y agobut that's literally what most chess players do anyway?
- Jensson 4y agoYes, they remember patterns, and then applies logic to make a valid move. ChatGPT remembers patterns, but didn't apply any logic to make valid moves. There is a difference there. ChatGPT does the pattern matching part, but not the logical part.