3 ms·

I'm curious, what kind of setup is required to train something like that? Can it be done with 4090 and how long would it take? 50M or 25M parameters? And what i

I'm curious, what kind of setup is required to train something like that? Can it be done with 4090 and how long would it take? 50M or 25M parameters? And what is the most parameters 4090 can do? This inspires few ideas.

anotherjesse 3y ago
More details in the previous blog post: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... > A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs. And from the paper: https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2403.15498 > The 25M parameter model took 72 hours to train on one RTX 3090 GPU. The 50M parameter model took 38 hours to train on four RTX 3090 GPUs. definitely inspiring :)
refulgentis 3y ago
I'm waiting for a build so I went fully down this trail. XD A1) Yes it can be done with a 4090. (2a). A2) 2 days. (4d/4e). B) Up to you, author did both and settled on 50M which they got to 1300 ELO. (note: they also did subsequent work with the same model to increase perf without further training) (2a). C) Mu: there's no limit as you can page in/out (3a, 3c), and its trivial to store in both RAM and VRAM. 30B fits in memory with 32 GB of RAM or VRAM. (3e) Starting from: 1. article: https://adamkarvonen.github.io/machine_learning/2024/03/20/chess-gpt-interventions.html https://adamkarvonen.github.io/machine_learning/2024/03/20/c... 2. second sentence links to post on training: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c.... 2a. "A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs" 2b. "The 50M parameter model played at 1300 Elo with 99.8% of its moves being legal within one day of training" 3. re: most parameters 4090 can do: 3a. understanding is there is no limitation, in that, you don't _need_ to have every parameter in memory at all times, either during training or inference. 3b. google "are amount of parameters in llm limited by vram size" 3c. go to /r/LocalLLaMa link: https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_are_the_variables_to_take_into_account_to/ https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_ar... (why? its a favorite of mine, believe its the closest you get to local training ppl talking in open space that's not discord) 3d. understanding in 3a is correct. 3e. "The model must fit in your RAM or VRAM, but you can split the model between them. With 32gb ram you could fit a 30b model" 4. 3090 vs. 4090 training speed 4a. google "rtx 3090 vs. 4090 llm training perf" 4b. wander through top reddit links. useful info, but a little too technical to share in a way that doesn't require explaining a lot. 4c. down the list: lamda labs link from october 2022: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep... 4d. 4090 is roughly 2x the perf on TransformerXL Large, the best match for transformers based large model, i.e. an LLM 4e. training took 4 3090-sdays for author. So 2 days.
- anotherjesse 3y ago
  Additionally instructions on training/inference on mac - https://github.com/adamkarvonen/nanoGPT https://github.com/adamkarvonen/nanoGPT > To sample on Mac, uncomment line 21 in sample.py. To train on Mac, rename train_shakespeare_char_mac.py to train_shakespeare_char.py The `mac` file changed several things - I decided to try running training with the original config file - changing device to mps / compile to false iter 100: loss 2.0268, time 815.43ms, mfu 3.24% iter 200: loss 1.8523, time 818.79ms, mfu 3.24% iter 300: loss 1.7799, time 823.05ms, mfu 3.23% iter 400: loss 1.6887, time 819.08ms, mfu 3.23% Training is ~4x slower than the speed reported on the original multi-GPU run: https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6?nw=nwuseradamkarvonen https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6... Not bad for an M2 studio which is running lots of other workloads at the same time
  - a_wild_dandan 3y ago
    I wonder if using Apple’s new MLX Python library (for training on unified memory systems) would yield significant gains.
    - nl 3y ago
      This (the device='mps' version) already uses the unified memory plus GPU on M-series Macs. It's possible MLX has some additional micro optimizations, but in general most people who have tried it out against hand-written MPS based training implementations haven't found great speed ups yet.

anotherjesse 3y ago

More details in the previous blog post: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... > A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs. And from the paper: https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2403.15498 > The 25M parameter model took 72 hours to train on one RTX 3090 GPU. The 50M parameter model took 38 hours to train on four RTX 3090 GPUs. definitely inspiring :)

refulgentis 3y ago

I'm waiting for a build so I went fully down this trail. XD A1) Yes it can be done with a 4090. (2a). A2) 2 days. (4d/4e). B) Up to you, author did both and settled on 50M which they got to 1300 ELO. (note: they also did subsequent work with the same model to increase perf without further training) (2a). C) Mu: there's no limit as you can page in/out (3a, 3c), and its trivial to store in both RAM and VRAM. 30B fits in memory with 32 GB of RAM or VRAM. (3e) Starting from: 1. article: https://adamkarvonen.github.io/machine_learning/2024/03/20/chess-gpt-interventions.html https://adamkarvonen.github.io/machine_learning/2024/03/20/c... 2. second sentence links to post on training: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c.... 2a. "A 50 million parameter GPT trained on 5 million games of chess learns to play at ~1300 Elo in one day on 4 RTX 3090 GPUs" 2b. "The 50M parameter model played at 1300 Elo with 99.8% of its moves being legal within one day of training" 3. re: most parameters 4090 can do: 3a. understanding is there is no limitation, in that, you don't _need_ to have every parameter in memory at all times, either during training or inference. 3b. google "are amount of parameters in llm limited by vram size" 3c. go to /r/LocalLLaMa link: https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_are_the_variables_to_take_into_account_to/ https://www.reddit.com/r/LocalLLaMA/comments/15j0mvm/what_ar... (why? its a favorite of mine, believe its the closest you get to local training ppl talking in open space that's not discord) 3d. understanding in 3a is correct. 3e. "The model must fit in your RAM or VRAM, but you can split the model between them. With 32gb ram you could fit a 30b model" 4. 3090 vs. 4090 training speed 4a. google "rtx 3090 vs. 4090 llm training perf" 4b. wander through top reddit links. useful info, but a little too technical to share in a way that doesn't require explaining a lot. 4c. down the list: lamda labs link from october 2022: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep... 4d. 4090 is roughly 2x the perf on TransformerXL Large, the best match for transformers based large model, i.e. an LLM 4e. training took 4 3090-sdays for author. So 2 days.

anotherjesse 3y ago
Additionally instructions on training/inference on mac - https://github.com/adamkarvonen/nanoGPT https://github.com/adamkarvonen/nanoGPT > To sample on Mac, uncomment line 21 in sample.py. To train on Mac, rename train_shakespeare_char_mac.py to train_shakespeare_char.py The `mac` file changed several things - I decided to try running training with the original config file - changing device to mps / compile to false iter 100: loss 2.0268, time 815.43ms, mfu 3.24% iter 200: loss 1.8523, time 818.79ms, mfu 3.24% iter 300: loss 1.7799, time 823.05ms, mfu 3.23% iter 400: loss 1.6887, time 819.08ms, mfu 3.23% Training is ~4x slower than the speed reported on the original multi-GPU run: https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6?nw=nwuseradamkarvonen https://wandb.ai/adam-karvonen/chess-gpt-batch/runs/zt5htyl6... Not bad for an M2 studio which is running lots of other workloads at the same time
- a_wild_dandan 3y ago
  I wonder if using Apple’s new MLX Python library (for training on unified memory systems) would yield significant gains.
  - nl 3y ago
    This (the device='mps' version) already uses the unified memory plus GPU on M-series Macs. It's possible MLX has some additional micro optimizations, but in general most people who have tried it out against hand-written MPS based training implementations haven't found great speed ups yet.