2 ms·
> 20GB RAM with Q4 quant. Closer to 25GB for the 4_K_M one how does this math work? are there rules of thumb that you guys know that the rest of us dont?
by swyx 2y ago
> 20GB RAM with Q4 quant. Closer to 25GB for the 4_K_M one
how does this math work? are there rules of thumb that you guys know that the rest of us dont?
- mekpro 2y agoAs a quick estimation, the size of q4 quantized model usually be around 60-70% of the model's parameter. You can preciselly check the quantized model size from .gguf files hosted in huggingface.