3 ms·
just finding the perfect spot between accuracy of the answers/available VRAM/tokens per second
by paoliniluis 7mo ago
just finding the perfect spot between accuracy of the answers/available VRAM/tokens per second
- tasuki 7mo agoOk, say I have 14GB VRAM. What is the tradeoff between using 9B with 8-bit params vs 27B with 3-bit params?
- causal 7mo ago3-bit 27B will almost certainly be better. 4-bits is usually the limit below-which you start to see more steep drop-offs, but you also get diminishing returns above 6-bits. So I'd still rather pack in more params at 3-bits. 9B will be faster, however.
- causal 7mo agoYou COULD even do Qwen3.5-35B-A3B-GGUF. UD-IQ3-XXS is only 13.1GB, which might outperform both in both intelligence and certainly speed (only 3B activated): https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF To accommodate cache you will need to offload a few feed-forward layers to the CPU. Will still be quite fast. Edit: Actually 27B does a little better than 35B on most benchmarks- 35B will still be much faster.
- andai 7mo agoSee also: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks