5 ms·
i was curious on what it takes to run this, the smallest ovh public cloud instance with gpu costs +500$/month before taxes
by Raed667 3y ago
i was curious on what it takes to run this, the smallest ovh public cloud instance with gpu costs +500$/month before taxes
- lionkor 3y agoPresumably some compute per hour service would make more sense for playing around with it?
- thelastparadise 3y agoCould it run on a 4x 3090 24GB rig? These can be built for about $4500 or less all-in. Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.
- mutex_man 3y agoYou could run it on a single high end GPU. I can run llama2's models ,(except 70b) on my 4080.
- 7moritz7 3y agoThis can run on 1x 2060S 8 GB
- ComputerGuru 3y agoWith what degree of quantization?
- 7moritz7 3y agoNone, just the default weights using ollama. It's fast too. 13b is where things get slow
- chriscappuccio 3y agodoes a 4x 3090 rig need nvswitch
- speedgoose 3y agoYou can run a 7B model on CPU relatively quickly. If you want to go faster, the best value in public clouds may be a rented Mac mini.
- objektif 3y agoDo you have any resources to read on how to host LLMs in general? I am looking for scaleable ways to host our own models. Thanks.
- speedgoose 3y agoSorry I haven’t followed the latest developments to run at scale since the summer. I don’t have concurrent users so llama.cpp or diffusers are good enough for me.