3 ms·
Ok, say I have 14GB VRAM. What is the tradeoff between using 9B with 8-bit params vs 27B with 3-bit params?
by tasuki 8mo ago
Ok, say I have 14GB VRAM. What is the tradeoff between using 9B with 8-bit params vs 27B with 3-bit params?
- causal 8mo ago3-bit 27B will almost certainly be better. 4-bits is usually the limit below-which you start to see more steep drop-offs, but you also get diminishing returns above 6-bits. So I'd still rather pack in more params at 3-bits. 9B will be faster, however.
- causal 8mo agoYou COULD even do Qwen3.5-35B-A3B-GGUF. UD-IQ3-XXS is only 13.1GB, which might outperform both in both intelligence and certainly speed (only 3B activated): https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF To accommodate cache you will need to offload a few feed-forward layers to the CPU. Will still be quite fast. Edit: Actually 27B does a little better than 35B on most benchmarks- 35B will still be much faster.
- andai 7mo agoSee also: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks