4 ms·
I doubt many people are using local setups for serious work. Even fine tuning Mixtral is 4xH100 for 4 days. Which is a ~$200k server currently. To fully train
by bearjaws 3y ago
I doubt many people are using local setups for serious work.
Even fine tuning Mixtral is 4xH100 for 4 days. Which is a ~$200k server currently.
To fully train, not just fine tune a small model, say Llama 2 7b you need over 128GiB of vram, so still multiple GPU territory, likely A100s or H100s.
This is all dependent upon the settings you use, increase the batch size and you will see even more memory utilization.
I believe a lot of people see these models running locally and assume training is similar, but it isn't.
- YetAnotherNick 3y agoWith qlora, 7b models could be fitted in 24 GB VRAM. And I have seen few folks doing it locally in discord servers I am part of.
- bearjaws 3y agoFor inference and fine tuning yes, not training.
- YetAnotherNick 3y agoFine tuning is training. Just done on smaller and higher quality dataset. For pretraining, RAM is the least of the issue. Even for smallest llama 2 they used 12 years of A100 GPU time.