3 ms·
3.8-flash-next fits on a single 6000 at Q4 if you offload the PLE. Crazy fast and still effective.
by lowbloodsugar 1mo ago
3.8-flash-next fits on a single 6000 at Q4 if you offload the PLE. Crazy fast and still effective.
- agile-gift0262 1mo agoSorry for the tangent, but how does Qwen3.8-flash-next compare to DeepSeek v4 Flash? I still haven't found the time to set it up, but I'm really happy with DeepSeek v4 Flash
- lowbloodsugar 1mo agoIt was the best model given my constraints (RTX PRO 6000 96gb + 256GB DDR4), when run against rust programming benchmarks. For Qwen3.8-flash-next NVFP4 and the latest vllm container, the PLE is 100GB of main ram, and everything else runs on the GPU with room for a total of 560k tokens (two full 262k conversations). DeepSeek has to offload a ton to the CPU and it performed worse than Qwen in absolute terms and was a lot slower (not usable). If you have enough room to run DeepSeek v4 Flash comfortably then you can likely run the Q8 of the qwen model.