21 ms·
Not that I understand the internals of current AI tech, but... I'd expect that an AI that has seen billions of chess positions, and the moves played in them, c
by BurningFrog 2y ago
Not that I understand the internals of current AI tech, but...
I'd expect that an AI that has seen billions of chess positions, and the moves played in them, can figure out the rules for legal moves without being told?
- rscho 2y agoStatistical 'AI' doesn't 'understand' anything, strictly speaking. It predicts a move with high probability, which could be legal or illegal.
- griomnib 2y agoLikewise with LLM you don’t know if it is truly in the “chess” branch of the statistical distribution or it is picking up something else entirely, like some arcane overlap of tokens. So much of the training data (eg common crawl, pile, Reddit) is dogshit, so it generates reheated dogshit.
- Helonomoto 2y agoYou generalize this without mentioning that there are LLMs which do not just use random 'dogshit'. Also what does a normal human do? It looks around how to move one random piece and it uses a very small dictionary / set of basic rules to move it. I do not remember me learning to count every piece and its options by looking up that rulebook. I learned to 'see' how i can move one type of chess piece. If a LLM uses only these piece moves on a mathematical level, it would do the same thing as i do. And yes there is also absolutly the option for an LLM to learn some kind of meta game.
- Helonomoto 2y agoHow do you define 'understand'? There is plenty of AI which learns the rules of games like Alpha Zero. LLMs might not have the architecture to 'learn', but it also might. If it optimizes all possible moves one chess peace can do (which is not that much to learn) it can easily only 'move' from one game set to another by this type of dictionary.
- rscho 2y agoUnderstanding a rules-based system (chess) means to be able to learn non-probabilistic rules (an abstraction over the concrete world). Humans are a mix of symbolic and probabilistic learning, allowing them to get a huge boost in performance by admitting rules. It doesn't mean a human will never make an illegal move, but it means a much smaller probability of illegal move based on less training data. Asymptotically, performance from humans and purely probabilistic systems converge. But that also means that in appropriate situations, humans are hugely more data-efficient.
- david-gpu 2y ago> in appropriate situations, humans are hugely more data-efficient After spending some years raising my children I gave up the notion that humans are data efficient. It takes a mind numbing amount of training to get them to learn the most basic skills.
- fragmede 2y agoThe illegal moves are interesting as it goes to "understanding". In children learning to play chess, how often do they try and make illegal moves? When first learning the game I remember that I'd lose track of all the things going on at once and try to make illegal moves, but eventually the rules became second nature and I stopped trying to make illegal moves. With an ELO of 1800, I'd expect ChatGPT not to make any illegal moves.
- sixfiveotwo 2y agoI think the article briefly touch on that topic at some point: > For one, gpt-3.5-turbo-instruct rarely suggests illegal moves, even in the late game. This requires “understanding” chess. If this doesn’t convince you, I encourage you to write a program that can take strings like 1. e4 d5 2. exd5 Qxd5 3. Nc3 and then say if the last move was legal. However, I can't say if LLMs fall in the "statistical AI" category.
- pvitz 2y agoA system that would just output the most probable tokens based on the text it was fed and trained on the games played by players with ratings greater than 1800 would certainly fail to output the right moves to totally unlikely board positions.
- Helonomoto 2y agoYes in theory it could. Depends on how it learns. Does it learn by memorization or by learning the rules. It depends on the architecture and the amount of 'pressure' you put on it to be more efficient or not.