3 ms·
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level che
by famouswaffles 1mo ago
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves.
Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
- bigstrat2003 1mo ago> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
- skydhash 1mo agoPretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
- willmarch 1mo agoIf you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn). Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
- rsfern 29d agothe discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
- willmarch 29d agoThat has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
- rsfern 29d agoMaybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)
- willmarch 27d agoYour child never messed up en passant (or had trouble understanding it in a real game), castled through or into a check, didn't see a discovered check after moving a piece, never got confused by how stalemate works?
- what 1mo ago> considering LLMs currently play better than a brand new human player would They’ve ingested all the literature on playing chess, a brand new human player has not.
- willmarch 29d agoYes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either). We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
- hi_im_greg_h 29d ago1. The LLMs have surely ingested hundreds if not thousands of books on chess. 2. The study (along with other posters here) show the models can’t even stick to following the rules of the game
- famouswaffles 1mo agoIf humans were actually intelligent, they wouldn't need to train and practice to play good chess. I mean, what level do you think people without any practice or training are ?
- bigstrat2003 29d agoOpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.
- cindyllm 29d ago[dead]
- diehunde 1mo agoExcept all these LLMs were already trained with hundreds of chess book and game databases and they still suck
- famouswaffles 29d agoIf all you do is read chess books, you'll be a shit player. Training and practice is what it takes to be great.
- diehunde 29d agoOh right. But if all you do is reading programming books you are an amazing programmer? Where is all the training and practice LLMs did to become so good at coding?
- famouswaffles 29d agoLLMs (and Humans) don't get really good from programming books lol. The training and practice is the actual code they predict and learn from in the process of predicting.
- Gregkion 29d agoThats just absolutly not true. A human being has general intelligence and needs A LOT of training and finetuning to become good in chess. And there is a relevant and significant difference between the expectation of an AGI and an ASI system.
- foldr 29d agoHumans don't need a lot of training and finite tuning to make only legal moves. An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves. An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
- thom 29d agoHow long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
- foldr 29d agoI don't know. My understanding is that current models will eventually fall into making illegal moves in longer chess games, and that no amount of prompting reliably gets them to stop doing so.