5 ms·
Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?
by nblgbg 2mo ago
Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?
- benxh 2mo agoDepends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus
- petu 2mo agoUnsloth one is gguf for llama.cpp (and some other on-device engines). So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
- 4chandaily 2mo agoRun the unsloth if you are using llama.cpp (GGUF) Run the one you linked if you are running vllm (safetensors)
- danielhanchen 2mo agoWe also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
- hadlock 2mo agoThis is the version we'll be testing on our rtx 6000 today! Thank you
- eblanshey 2mo agoWhy not just run FP8 on vLLM with that much vRAM? It's plenty fast.
- hadlock 2mo agoFor high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number. I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
- eblanshey 2mo agoI see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?
- hadlock 2mo agoI didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.
- satvikpendem 2mo agoUnsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
- danielhanchen 2mo agoThat wasn't our problem right? Gemma officially updated tool calling which we adopted
- ycui7 2mo agoif you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop