3 ms·
>initial server cost of $3,800 Not following? Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay. 700 for the card and another 500 fo
by Havoc 2y ago
>initial server cost of $3,800
Not following?
Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay. 700 for the card and another 500 for some 2nd hand basic gaming rig.
Plus you don't need a 4 slot PCIE mobo. Plus it's a gen4 pcie card (vs gen3). Plus skipping the complexity of multi-GPU. And wouldn't be surprised if it ends up faster too (everything in one GPU tends to be much faster in my experience, plus 3090 is just organically faster 1:1)
Or if you're feeling extra spicy you can do same on a 7900XTX (inference works fine on those & it's likely that there will be big optimisation gains in next months).
- Sohcahtoa82 2y ago> Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay Someone correct me if I'm wrong, but I've always thought you needed enough VRAM to have at least double the model size so that the GPU has enough VRAM for the calculated values from the model. So that 17 GB model requires 34 GB of RAM. Though you can quantize to fp8/int8 with surprisingly little negative effect and then run that 17 GB model with 17 GB of VRAM.
- jokethrowaway 2y agoNo, you don't need that much Here is a calculator (if you have a GPU you want to use EXL2, otherwise GGUF) https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... Also model quantisation goes a long way with surprisingly little loss in quality.
- wkat4242 2y agoYou do need more if you use larger context sizes though. It can really blow up to multiple times the model size even for 128k context. Edit: Oh I see that calculator you linked shows that too. My information was more trial and error, thanks for the calculator link.