4 ms·
Could it run on a 4x 3090 24GB rig? These can be built for about $4500 or less all-in. Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.
by thelastparadise 3y ago
Could it run on a 4x 3090 24GB rig?
These can be built for about $4500 or less all-in.
Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.
- mutex_man 3y agoYou could run it on a single high end GPU. I can run llama2's models ,(except 70b) on my 4080.
- 7moritz7 3y agoThis can run on 1x 2060S 8 GB
- ComputerGuru 3y agoWith what degree of quantization?
- 7moritz7 3y agoNone, just the default weights using ollama. It's fast too. 13b is where things get slow
- chriscappuccio 3y agodoes a 4x 3090 rig need nvswitch