4 ms·

>ps, but for smaller models, such as the 7B a single GPU suffices If I wanna do this, what GPU would I need?? I have a 3060 TI (laptop one) and i9 with 16 gigs

>ps, but for smaller models, such as the 7B a single GPU suffices If I wanna do this, what GPU would I need?? I have a 3060 TI (laptop one) and i9 with 16 gigs of ram. I don't have AWS quota, or quota in GCP either. have heard of paperspace, but I want to quickly get started with fine tuning Mistral, cause we are planning to use some of their models for an engagement I am working on.

qayxc 2y ago
Check out this site: https://www.hardware-corner.net/llm-database/Mistral/ https://www.hardware-corner.net/llm-database/Mistral/ They list hardware requirements by model and you can select your VRAM and system memory to filter available models.
- elpocko 2y ago
  Unless I'm missing something - this is for inference only, using quantized models. Fine-tuning usually has much higher requirements.
semi-extrinsic 2y ago
Unless you have zero budget, I would strongly suggest going up to a "gaming" desktop computer. A gaming GPU can put out 300 W of heat no problem. If your laptop GPU did the same it would melt, it's going to be limited to around 100 W. And heat dissipation is directly proportionaly to speed. Plus in a desktop, you have the ability to upgrade to faster GPU or even go for multiple GPUs. Caveat: such machines, especially multi-GPU, are loud and produce enough heat to warm up a whole room quickly. And cloud will likely be cheaper if you believe you will not spend more than 10% of the next couple of years running the GPU at full blast.
- washadjeffmad 2y ago
  >heat dissipation is directly proportionaly to speed. Power limiting Nvidia GPUs up to around 60% of max TDP can offer a substantial reduction to heat and noise with minimal performance loss, but agreed on everything else.
- rpastuszak 2y ago
  I'm using an M1 MBA and was considering getting an eGPU instead of buying a PC, would you advise against it? What variables would you consider?
  - addandsubtract 2y ago
    There are no Nvidia drivers for Mac.
Narciss 2y ago
Try out openpipe. I am working with it at my current company and have seen some good results.
aunty_helen 2y ago
You can get a hetzner gpu box for 184€ a month. My company has been fine tuning mistal and llama3 on the rtx4000 these have. It is slightly limited in ram at only 20gb but we found dropping the quantisation has helped with larger input token counts. They also now do hourly rental.

qayxc 2y ago

Check out this site: https://www.hardware-corner.net/llm-database/Mistral/ https://www.hardware-corner.net/llm-database/Mistral/ They list hardware requirements by model and you can select your VRAM and system memory to filter available models.

elpocko 2y ago
Unless I'm missing something - this is for inference only, using quantized models. Fine-tuning usually has much higher requirements.

semi-extrinsic 2y ago

Unless you have zero budget, I would strongly suggest going up to a "gaming" desktop computer. A gaming GPU can put out 300 W of heat no problem. If your laptop GPU did the same it would melt, it's going to be limited to around 100 W. And heat dissipation is directly proportionaly to speed. Plus in a desktop, you have the ability to upgrade to faster GPU or even go for multiple GPUs. Caveat: such machines, especially multi-GPU, are loud and produce enough heat to warm up a whole room quickly. And cloud will likely be cheaper if you believe you will not spend more than 10% of the next couple of years running the GPU at full blast.

washadjeffmad 2y ago
>heat dissipation is directly proportionaly to speed. Power limiting Nvidia GPUs up to around 60% of max TDP can offer a substantial reduction to heat and noise with minimal performance loss, but agreed on everything else.
rpastuszak 2y ago
I'm using an M1 MBA and was considering getting an eGPU instead of buying a PC, would you advise against it? What variables would you consider?
- addandsubtract 2y ago
  There are no Nvidia drivers for Mac.

Narciss 2y ago

Try out openpipe. I am working with it at my current company and have seen some good results.

aunty_helen 2y ago

You can get a hetzner gpu box for 184€ a month. My company has been fine tuning mistal and llama3 on the rtx4000 these have. It is slightly limited in ram at only 20gb but we found dropping the quantisation has helped with larger input token counts. They also now do hourly rental.