5 ms·

Manipulating Chess-GPT's World Model

119 points by seraine 3y ago

seraine 3y ago
The code for this is located here: https://github.com/adamkarvonen/chess_llm_interpretability https://github.com/adamkarvonen/chess_llm_interpretability
dang 3y ago
Recent and related: Chess-GPT's Internal World Model - https://news.ycombinator.com/item?id=38893456 https://news.ycombinator.com/item?id=38893456 - Jan 2024 (103 comments)
- patresh 3y ago
  Another related one from last year based on the Othello game (cited in the above paper) : Do Large Language Models learn world models or just surface statistics? - https://news.ycombinator.com/item?id=34474043 https://news.ycombinator.com/item?id=34474043 - Jan 2023 (174 comments)
mxwsn 3y ago
I'd be curious to see which moves gain or lose the most probability when performing the skill intervention. Or the correlation between predicted move probabilities and engine score for moves, before and after skill intervention.
mewpmewp2 3y ago
I'm curious, what kind of setup is required to train something like that? Can it be done with 4090 and how long would it take? 50M or 25M parameters? And what is the most parameters 4090 can do? This inspires few ideas.
- anotherjesse 3y ago
  More details in the previous blog post: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... > A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs. And from the paper: https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2403.15498 > The 25M parameter model took 72 hours to train on one RTX 3090 GPU. The 50M parameter model took 38 hours to train on four RTX 3090 GPUs. definitely inspiring :)
- refulgentis 3y ago
  I'm waiting for a build so I went fully down this trail. XD A1) Yes it can be done with a 4090. (2a). A2) 2 days. (4d/4e). B) Up to you, author did both and settled on 50M which they got to 1300 ELO. (note: they also did subsequent work with the same model to increase perf without further training) (2a). C) Mu: there's no limit as you can page in/out (3a, 3c), and its trivial to store in both RAM and VRAM. 30B fits in memory with 32 GB of RAM or VRAM. (3e) Starting from: 1. article: https://adamkarvonen.github.io/machine_learning/2024/03/20/chess-gpt-interventions.html https://adamkarvonen.github.io/machine_learning/2024/03/20/c... 2. second sentence links to post on training: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c.... 2a. "A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs" 2b. "The 50M parameter model played at 1300 Elo with 99.8% of its moves being legal within one day of training" 3. re: most parameters 4090 can do: 3a. understanding is there is no limitation, in that, you don't _need_ to have every parameter in memory at all times, either during training or inference. 3b. google "are amount of parameters in llm limited by vram size" 3c. go to /r/LocalLLaMa link: https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_are_the_variables_to_take_into_account_to/ https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_ar... (why? its a favorite of mine, believe its the closest you get to local training ppl talking in open space that's not discord) 3d. understanding in 3a is correct. 3e. "The model must fit in your RAM or VRAM, but you can split the model between them. With 32gb ram you could fit a 30b model" 4. 3090 vs. 4090 training speed 4a. google "rtx 3090 vs. 4090 llm training perf" 4b. wander through top reddit links. useful info, but a little too technical to share in a way that doesn't require explaining a lot. 4c. down the list: lamda labs link from october 2022: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep... 4d. 4090 is roughly 2x the perf on TransformerXL Large, the best match for transformers based large model, i.e. an LLM 4e. training took 4 3090-sdays for author. So 2 days.
  - anotherjesse 3y ago
    Additionally instructions on training/inference on mac - https://github.com/adamkarvonen/nanoGPT https://github.com/adamkarvonen/nanoGPT > To sample on Mac, uncomment line 21 in sample.py. To train on Mac, rename train_shakespeare_char_mac.py to train_shakespeare_char.py The `mac` file changed several things - I decided to try running training with the original config file - changing device to mps / compile to false iter 100: loss 2.0268, time 815.43ms, mfu 3.24% iter 200: loss 1.8523, time 818.79ms, mfu 3.24% iter 300: loss 1.7799, time 823.05ms, mfu 3.23% iter 400: loss 1.6887, time 819.08ms, mfu 3.23% Training is ~4x slower than the speed reported on the original multi-GPU run: https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6?nw=nwuseradamkarvonen https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6... Not bad for an M2 studio which is running lots of other workloads at the same time
Der_Einzige 3y ago
All of the related work, such as activation/representation engineering, and control/steering vectors is also really neat! You can play with steering vectors within oobabooga now: https://github.com/Hellisotherpeople/llm_steer-oobabooga https://github.com/Hellisotherpeople/llm_steer-oobabooga
jprival 3y ago
Reminds me tangentially of the study that found that strong chess players were much worse at recalling randomized positions than realistic ones: https://link.springer.com/content/pdf/10.3758/BF03200937.pdf https://link.springer.com/content/pdf/10.3758/BF03200937.pdf I’m wary of inviting the too-easy “it’s just like how it works for people!” comparison, but the implied context and history of a game state seems to be important in processing it.
- pvg 3y ago
  Is it context and history or just that real game positions have much lower entropy? Most people can easily remember a sentence after a quick look, a similar-length string of random characters a lot less so.
  - 613style 3y ago
    I think this is correct. I play Go at a strong amateur level, and on the 19x19 board, I can often fully recall 200+ move games against "normal" opponents, because the moves tell a story that makes sense. But if I play someone much weaker or much stronger who makes a lot of unexpected moves, I'll have trouble remembering those moments.
- deleted 3y ago
  [deleted]
elif 3y ago
> had Chess-GPT play against Stockfish with this random initialization. Its performance plummeted. The larger 50 million parameter model’s win rate dropped from 70% to 17%. Beating stockfish 17% of the time is still incredible for any engine. and being trained on human moves rather than engine moves will make this so much more annoying to analyze cheating...
- jsmith99 3y ago
  Article says: ‘Chess-GPT also played chess well, with the best model playing at approximately 1500 Elo.’ So I’m guessing this wasn’t full strength stockfish.
  - noSyncCloud 3y ago
    It notes in the blog post that it was SF level 0.
  - jxy 3y ago
    The previous blog post [0] mentioned they were using stockfish with 0.1 seconds per move. [0] https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
- goatlover 3y ago
  It would be impressive if it was playing Stockfish at a grandmaster level. But it looks like it's lower-level. Still impressive that it can learn to play decently at some level. Doubt it will ever be competitive with top engine levels.
- seraine 3y ago
  I updated the article to also mention this in the second blog post. All games were against Stockfish level 0, with nodes searched limited to 100,000 rather than a time based search limit to limit variability due to different processors.
  - afro88 3y ago
QuantumG 3y ago
Anyone got more?
willd13 3y ago
I would be really interested to understand the efficiency of this learning vs a more traditional approach. If you have two models - ChessGPT and a traditional chess-playing ML model - and train them on the same number of games, which reaches a higher level of chess-playing skill?
mycall 3y ago
I look forward when AI intelligently starts combining games for us, i.e. ChessBattleShip
mvkel 3y ago
Curious how this performs against chatgpt with few-shot prompting. More and more evidence recently has shown that fine tunes/retrieval don't measurably improve performance over few-shot, but with this being such a specific domain/skill, I'd be surprised if it didn't far outperform a few-shot approach.
- kromem 3y ago
  It's also not a fine tune. It's a pretrained toy model in two days on consumer hardware.
kromem 3y ago
Two thoughts regarding the new work: (1) I'd consider normalizing the performance data for the random cases against another chess program with similar performance under normal conditions. It may be that introducing 20 random moves to the start of a game biases all players towards a 50/50 win outcome. So the sub-50 performance may not reflect a failure of flipping the "don't suck" switch, but simply good performance in a more average outcome scenario. It'd be interesting to see if Chess-GPT's relative performance against other chess programs in the random scenario was better than its relative performance in the normal case. (2) The 'fuzziness' of the board positions you found when removing the pawn makes complete sense given one of the nuanced findings in Hazineh, et al Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT (2023) - specifically the finding that it was encoding representations for board configuration and not just pieces (in that case three stones in a row). It may be that piecemeal removal of a piece disrupted patterns of how games normally flow which it had learned, and as such there was greater uncertainty than the original board state. A similar issue may be at hand with the random 20 moves to start, and I'd be curious what the confidence of the board state was when starting off 20 random moves in and if that confidence stabilized as the game went on from there. Overall really cool update! And bigger picture, the prospects of essentially flipping an internalized skill vector for larger models to bias them back away from their regression to the mean is particularly exciting.

119 points by seraine 3y ago

The code for this is located here: https://github.com/adamkarvonen/chess_llm_interpretability https://github.com/adamkarvonen/chess_llm_interpretability

dang 3y ago

Recent and related: Chess-GPT's Internal World Model - https://news.ycombinator.com/item?id=38893456 https://news.ycombinator.com/item?id=38893456 - Jan 2024 (103 comments)

patresh 3y ago
Another related one from last year based on the Othello game (cited in the above paper) : Do Large Language Models learn world models or just surface statistics? - https://news.ycombinator.com/item?id=34474043 https://news.ycombinator.com/item?id=34474043 - Jan 2023 (174 comments)

mxwsn 3y ago

I'd be curious to see which moves gain or lose the most probability when performing the skill intervention. Or the correlation between predicted move probabilities and engine score for moves, before and after skill intervention.

mewpmewp2 3y ago

I'm curious, what kind of setup is required to train something like that? Can it be done with 4090 and how long would it take? 50M or 25M parameters? And what is the most parameters 4090 can do? This inspires few ideas.

anotherjesse 3y ago
More details in the previous blog post: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... > A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs. And from the paper: https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2403.15498 > The 25M parameter model took 72 hours to train on one RTX 3090 GPU. The 50M parameter model took 38 hours to train on four RTX 3090 GPUs. definitely inspiring :)
refulgentis 3y ago
I'm waiting for a build so I went fully down this trail. XD A1) Yes it can be done with a 4090. (2a). A2) 2 days. (4d/4e). B) Up to you, author did both and settled on 50M which they got to 1300 ELO. (note: they also did subsequent work with the same model to increase perf without further training) (2a). C) Mu: there's no limit as you can page in/out (3a, 3c), and its trivial to store in both RAM and VRAM. 30B fits in memory with 32 GB of RAM or VRAM. (3e) Starting from: 1. article: https://adamkarvonen.github.io/machine_learning/2024/03/20/chess-gpt-interventions.html https://adamkarvonen.github.io/machine_learning/2024/03/20/c... 2. second sentence links to post on training: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c.... 2a. "A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs" 2b. "The 50M parameter model played at 1300 Elo with 99.8% of its moves being legal within one day of training" 3. re: most parameters 4090 can do: 3a. understanding is there is no limitation, in that, you don't _need_ to have every parameter in memory at all times, either during training or inference. 3b. google "are amount of parameters in llm limited by vram size" 3c. go to /r/LocalLLaMa link: https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_are_the_variables_to_take_into_account_to/ https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_ar... (why? its a favorite of mine, believe its the closest you get to local training ppl talking in open space that's not discord) 3d. understanding in 3a is correct. 3e. "The model must fit in your RAM or VRAM, but you can split the model between them. With 32gb ram you could fit a 30b model" 4. 3090 vs. 4090 training speed 4a. google "rtx 3090 vs. 4090 llm training perf" 4b. wander through top reddit links. useful info, but a little too technical to share in a way that doesn't require explaining a lot. 4c. down the list: lamda labs link from october 2022: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep... 4d. 4090 is roughly 2x the perf on TransformerXL Large, the best match for transformers based large model, i.e. an LLM 4e. training took 4 3090-sdays for author. So 2 days.
- anotherjesse 3y ago
  Additionally instructions on training/inference on mac - https://github.com/adamkarvonen/nanoGPT https://github.com/adamkarvonen/nanoGPT > To sample on Mac, uncomment line 21 in sample.py. To train on Mac, rename train_shakespeare_char_mac.py to train_shakespeare_char.py The `mac` file changed several things - I decided to try running training with the original config file - changing device to mps / compile to false iter 100: loss 2.0268, time 815.43ms, mfu 3.24% iter 200: loss 1.8523, time 818.79ms, mfu 3.24% iter 300: loss 1.7799, time 823.05ms, mfu 3.23% iter 400: loss 1.6887, time 819.08ms, mfu 3.23% Training is ~4x slower than the speed reported on the original multi-GPU run: https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6?nw=nwuseradamkarvonen https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6... Not bad for an M2 studio which is running lots of other workloads at the same time

Der_Einzige 3y ago

All of the related work, such as activation/representation engineering, and control/steering vectors is also really neat! You can play with steering vectors within oobabooga now: https://github.com/Hellisotherpeople/llm_steer-oobabooga https://github.com/Hellisotherpeople/llm_steer-oobabooga

jprival 3y ago

Reminds me tangentially of the study that found that strong chess players were much worse at recalling randomized positions than realistic ones: https://link.springer.com/content/pdf/10.3758/BF03200937.pdf https://link.springer.com/content/pdf/10.3758/BF03200937.pdf I’m wary of inviting the too-easy “it’s just like how it works for people!” comparison, but the implied context and history of a game state seems to be important in processing it.

pvg 3y ago
Is it context and history or just that real game positions have much lower entropy? Most people can easily remember a sentence after a quick look, a similar-length string of random characters a lot less so.
- 613style 3y ago
  I think this is correct. I play Go at a strong amateur level, and on the 19x19 board, I can often fully recall 200+ move games against "normal" opponents, because the moves tell a story that makes sense. But if I play someone much weaker or much stronger who makes a lot of unexpected moves, I'll have trouble remembering those moments.
deleted 3y ago
[deleted]

elif 3y ago

> had Chess-GPT play against Stockfish with this random initialization. Its performance plummeted. The larger 50 million parameter model’s win rate dropped from 70% to 17%. Beating stockfish 17% of the time is still incredible for any engine. and being trained on human moves rather than engine moves will make this so much more annoying to analyze cheating...

jsmith99 3y ago
Article says: ‘Chess-GPT also played chess well, with the best model playing at approximately 1500 Elo.’ So I’m guessing this wasn’t full strength stockfish.
- noSyncCloud 3y ago
  It notes in the blog post that it was SF level 0.
- jxy 3y ago
  The previous blog post [0] mentioned they were using stockfish with 0.1 seconds per move. [0] https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
goatlover 3y ago
It would be impressive if it was playing Stockfish at a grandmaster level. But it looks like it's lower-level. Still impressive that it can learn to play decently at some level. Doubt it will ever be competitive with top engine levels.
seraine 3y ago
I updated the article to also mention this in the second blog post. All games were against Stockfish level 0, with nodes searched limited to 100,000 rather than a time based search limit to limit variability due to different processors.
- afro88 3y ago

QuantumG 3y ago

Anyone got more?

willd13 3y ago

I would be really interested to understand the efficiency of this learning vs a more traditional approach. If you have two models - ChessGPT and a traditional chess-playing ML model - and train them on the same number of games, which reaches a higher level of chess-playing skill?

mycall 3y ago

I look forward when AI intelligently starts combining games for us, i.e. ChessBattleShip

mvkel 3y ago

Curious how this performs against chatgpt with few-shot prompting. More and more evidence recently has shown that fine tunes/retrieval don't measurably improve performance over few-shot, but with this being such a specific domain/skill, I'd be surprised if it didn't far outperform a few-shot approach.

kromem 3y ago
It's also not a fine tune. It's a pretrained toy model in two days on consumer hardware.

kromem 3y ago

Two thoughts regarding the new work: (1) I'd consider normalizing the performance data for the random cases against another chess program with similar performance under normal conditions. It may be that introducing 20 random moves to the start of a game biases all players towards a 50/50 win outcome. So the sub-50 performance may not reflect a failure of flipping the "don't suck" switch, but simply good performance in a more average outcome scenario. It'd be interesting to see if Chess-GPT's relative performance against other chess programs in the random scenario was better than its relative performance in the normal case. (2) The 'fuzziness' of the board positions you found when removing the pawn makes complete sense given one of the nuanced findings in Hazineh, et al Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT (2023) - specifically the finding that it was encoding representations for board configuration and not just pieces (in that case three stones in a row). It may be that piecemeal removal of a piece disrupted patterns of how games normally flow which it had learned, and as such there was greater uncertainty than the original board state. A similar issue may be at hand with the random 20 moves to start, and I'd be curious what the confidence of the board state was when starting off 20 random moves in and if that confidence stabilized as the game went on from there. Overall really cool update! And bigger picture, the prospects of essentially flipping an internalized skill vector for larger models to bias them back away from their regression to the mean is particularly exciting.