3 ms·
I'm getting 30 t/s on RTX 4090D (using 42 out of 48GB VRAM) with UD-Q6_K_XL https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/7 https://huggingface.c
by SlavikCA 5mo ago
I'm getting 30 t/s on RTX 4090D (using 42 out of 48GB VRAM) with UD-Q6_K_XL
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/7 https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/...
- maxloh 5mo agoI thought Q4_K_M is the standard. Why did you choose the 6-bit variant? Does it generate better input?
- SlavikCA 5mo agoThere is no standard. The higher quantization - the better results, but more memory is needed. Q8 is the best.
- SV_BubbleTime 5mo agoFP32 is best, although I wonder if there isn’t something better I don’t know about. Q8 is for the most part equal to FP16 in practical terms by being smart about what is quantized, but iirc always slower than FP16 and FP8.