4 ms·
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to of
by Anonyneko 2mo ago
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to offload a lot to 64GB RAM and a beefy CPU (but still a CPU)
- NekkoDroid 2mo agoI personally run the Q6 quant on my RX 9070 XT (16GB VRAM). On r/LocalLlama there was a post recently as well, which talked about the degradation of different quants (for the 27B version)[0] [0]: https://www.reddit.com/r/LocalLLaMA/comments/1vef79c/quantization_hurts_knowledge_nonlinearly_qwen36/ https://www.reddit.com/r/LocalLLaMA/comments/1vef79c/quantiz...