4 ms·
> had Chess-GPT play against Stockfish with this random initialization. Its performance plummeted. The larger 50 million parameter model’s win rate dropped from
by elif 3y ago
> had Chess-GPT play against Stockfish with this random initialization. Its performance plummeted. The larger 50 million parameter model’s win rate dropped from 70% to 17%.
Beating stockfish 17% of the time is still incredible for any engine.
and being trained on human moves rather than engine moves will make this so much more annoying to analyze cheating...
- jsmith99 3y agoArticle says: ‘Chess-GPT also played chess well, with the best model playing at approximately 1500 Elo.’ So I’m guessing this wasn’t full strength stockfish.
- noSyncCloud 3y agoIt notes in the blog post that it was SF level 0.
- jxy 3y agoThe previous blog post [0] mentioned they were using stockfish with 0.1 seconds per move. [0] https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
- goatlover 3y agoIt would be impressive if it was playing Stockfish at a grandmaster level. But it looks like it's lower-level. Still impressive that it can learn to play decently at some level. Doubt it will ever be competitive with top engine levels.
- seraine 3y agoI updated the article to also mention this in the second blog post. All games were against Stockfish level 0, with nodes searched limited to 100,000 rather than a time based search limit to limit variability due to different processors.
- afro88 3y agoI googled stockfish "level 0" and it came up with no results. What do you mean by level 0?
- seraine 3y agoThe Stockfish program can be set to play at strength level 0-20. Estimates of the levels' Elo is provided here: https://github.com/official-stockfish/Stockfish/commit/a08b8d4 https://github.com/official-stockfish/Stockfish/commit/a08b8...
- andrepd 3y agoIt means it plays several times worse than a 1960s chess engine.
- quadrature 3y agoHave you tried performing a search using Chess-GPT ?.
- swyx 3y agoi think i dont understand what stockfish does then. doesn't it play the basically perfect move at each step? how is it possible to beat stockfish 70% of the time? with a naive GPT type thing?
- deleted 3y ago[deleted]
- makeset 3y agoChess is not a fully "solved" game, so the actual perfect move is not generally known. Like any other engine, Stockfish just tries its best and is not infallible.
- elif 3y agoTechnically, the tablebases for 7-piece chess are fully solved, and since we are talking about games starting from partial board states, you could trivially compare both engines to the "perfect" move for a huge number of board states.
- canadianfella 3y agoThat’s not the right answer. The real answer is it was playing against a stockfish at a low difficulty setting.
- swyx 3y agowell that's much more palatable, haha
- nl 3y ago> Doesn't it play the basically perfect move at each step? "the basically perfect move" has a lot of implications here. Because chess isn't a solved game a seemingly "perfect move" in a game involving two 2000 ELO rated players might not seem perfect to a 3000 ELO rated player. (To be clear: the only actual perfect moves are ones that all trees lead to a checkmate, but in general use "perfect move" means roughly the best possible move to play) In this case Stockfish was set to a lower setting which corresponds to 1320 ELO.