3 ms·
We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
by danielhanchen 2mo ago
We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
- hadlock 2mo agoThis is the version we'll be testing on our rtx 6000 today! Thank you
- eblanshey 2mo agoWhy not just run FP8 on vLLM with that much vRAM? It's plenty fast.
- hadlock 2mo agoFor high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number. I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
- eblanshey 2mo agoI see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?
- hadlock 2mo agoI didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.