4 ms·
> Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay Someone correct me if I'm wrong, but I've always thought you needed enough VRAM t
by Sohcahtoa82 2y ago
> Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay
Someone correct me if I'm wrong, but I've always thought you needed enough VRAM to have at least double the model size so that the GPU has enough VRAM for the calculated values from the model. So that 17 GB model requires 34 GB of RAM.
Though you can quantize to fp8/int8 with surprisingly little negative effect and then run that 17 GB model with 17 GB of VRAM.
- jokethrowaway 2y agoNo, you don't need that much Here is a calculator (if you have a GPU you want to use EXL2, otherwise GGUF) https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... Also model quantisation goes a long way with surprisingly little loss in quality.
- wkat4242 2y agoYou do need more if you use larger context sizes though. It can really blow up to multiple times the model size even for 128k context. Edit: Oh I see that calculator you linked shows that too. My information was more trial and error, thanks for the calculator link.