4 ms·
No, you don't need that much Here is a calculator (if you have a GPU you want to use EXL2, otherwise GGUF) https://huggingface.co/spaces/NyxKrage/LLM-Model-VRA
by jokethrowaway 2y ago
No, you don't need that much
Here is a calculator (if you have a GPU you want to use EXL2, otherwise GGUF)
https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul...
Also model quantisation goes a long way with surprisingly little loss in quality.
- wkat4242 2y agoYou do need more if you use larger context sizes though. It can really blow up to multiple times the model size even for 128k context. Edit: Oh I see that calculator you linked shows that too. My information was more trial and error, thanks for the calculator link.