3 ms·
With qlora, 7b models could be fitted in 24 GB VRAM. And I have seen few folks doing it locally in discord servers I am part of.
by YetAnotherNick 3y ago
With qlora, 7b models could be fitted in 24 GB VRAM. And I have seen few folks doing it locally in discord servers I am part of.
- bearjaws 3y agoFor inference and fine tuning yes, not training.
- YetAnotherNick 3y agoFine tuning is training. Just done on smaller and higher quality dataset. For pretraining, RAM is the least of the issue. Even for smallest llama 2 they used 12 years of A100 GPU time.