13 ms·
For some reason I thought fine tuning is not possible without specialized hardware (A100 / H100). Where can I learn more about hardware requirements for fine tu
by gigel82 3y ago
For some reason I thought fine tuning is not possible without specialized hardware (A100 / H100). Where can I learn more about hardware requirements for fine tuning on consumer GPUs?
- dmakian 3y agoThere is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft https://github.com/huggingface/peft)
- mmcwilliams 3y agoDefinitely possible on even older off-the-shelf hardware. I use 24GB 4090s for 13b-sized models and have even used 12GB Titans for 7b models, admittedly at much slower rates.
- viraptor 3y agoYou can also use Apple silicon for this: https://www.reddit.com/r/LocalLLaMA/comments/15y9m64/fine_tuningggml_quantiziation_on_apple_silicon/ https://www.reddit.com/r/LocalLLaMA/comments/15y9m64/fine_tu...
- gigel82 3y agoI have a 3080Ti with 12Gb VRAM and would like to try fine tuning the same Mistral 7B model (which I found incredibly potent). Any tips on how to get started?