5 ms·
For those interested, made some Dynamic Unsloth GGUFs for local deployment at https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF https://huggingface.co/unslot
by danielhanchen 8mo ago
For those interested, made some Dynamic Unsloth GGUFs for local deployment at https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF and made a guide on using Claude Code / Codex locally: https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next
- ranger_danger 8mo agoWhat is the difference between the UD and non-UD files?
- danielhanchen 8mo agoUD stands for "Unsloth-Dynamic" which upcasts important layers to higher bits. Non UD is just standard llama.cpp quants. Both still use our calibration dataset.
- CamperBob2 8mo agoPlease consider authoring a single, straightforward introductory-level page somewhere that explains what all the filename components mean, and who should use which variants. The green/yellow/red indicators for different levels of hardware support are really helpful, but far from enough IMO.
- danielhanchen 8mo agoOh good idea! In general UD-Q4_K_XL (Unsloth Dynamic 4bits Extra Large) is what I generally recommend for most hardware - MXFP4_MOE is also ok
- Keats 8mo agoIs there some indication on how the different bit quantization affect performance? IE I have a 5090 + 96GB so I want to get the best possible model but I don't care about getting 2% better perf if I only get 5 tok/s.
- mirekrusin 8mo agoIt takes download time + 1 minute to test speed yourself, you can try different quants, it's hard to write down a table because it depends on your system ie. ram clock etc. if you go out of gpu. I guess it would make sense to have something like max context size/quants that fit fully on common configs with gpus, dual gpus, unified ram on mac etc.
- Keats 8mo agoTesting speed is easy yes, I'm mostly wondering about the quality difference between Q6 vs Q8_K_XL for example.
- danielhanchen 8mo agoI haven't done benchmarking yet (plan to do them), but it should be similar to our post on DeepSeek-V3.1 Dynamic GGUFs: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- segmondy 8mo agoThe green/yellow/red indicators are based on what you set for your hardware on huggingface.
- ranger_danger 8mo agoWhat is your definition of "important" in this context?
- danielhanchen 8mo agoOh we wrote about it here: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- binsquare 8mo agoHow did you do it so fast? Great work as always btw!
- danielhanchen 8mo agoThanks! :) We're early access partners with them!
- genpfault 8mo agoNice! Getting ~39 tok/s @ ~60% GPU util. (~170W out of 303W per nvtop). System info: $ ./llama-server --version ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat version: 7897 (3dd95914d) built with GNU 11.4.0 for Linux x86_64 llama.cpp command-line: $ ./llama-server --host 0.0.0.0 --port 2000 --no-warmup \ -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \ --jinja --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 --fit on \ --ctx-size 32768
- halcyonblue 8mo agoWhat am I missing here? I thought this model needs 46GB of unified memory for 4-bit quant. Radeon RX 7900 XTX has 24GB of memory right? Hoping to get some insight, thanks in advance!
- coder543 8mo agoMoEs can be efficiently split between dense weights (attention/KV/etc) and sparse (MoE) weights. By running the dense weights on the GPU and offloading the sparse weights to slower CPU RAM, you can still get surprisingly decent performance out of a lot of MoEs. Not as good as running the entire thing on the GPU, of course.
- danielhanchen 8mo agoSuper cool! Also with `--fit on` you don't need `--ctx-size 32768` technically anymore - llama-server will auto determine the max context size!
- genpfault 8mo agoNifty, thanks for the heads-up!
- lnenad 8mo agoThanks to you I decided to give it a go as well (didn't think I'd be able to run it on 7900xtx) and I must say it's awesome for a local model. More than capable for more straightforward stuff. It uses full VRAM and about 60GBs of RAM, but runs at about 10tok/s and is *very* usable.
- bityard 8mo agoHi Daniel, I've been using some of your models on my Framework Desktop at home. Thanks for all that you do. Asking from a place of pure ignorance here, because I don't see the answer on HF or in your docs: Why would I (or anyone) want to run this instead of Qwen3's own GGUFs?
- danielhanchen 8mo agoThanks! Oh Qwen3's own GGUFs also works, but ours are dynamically quantized and calibrated with a reasonably large diverse dataset, whilst Qwen's ones are not - see https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
- bityard 8mo agoI've read that page before and although it all certainly sounds very impressive, I'm not an AI researcher. What's the actual goal of dynamic quantization? Does it make the model more accurate? Faster? Smaller?
- CamperBob2 8mo agoGood results with your Q8_0 version on 96GB RTX 6000 Blackwell. It one-shotted the Flappy Bird game and also wrote a good Wordle clone in four shots, all at over 60 tps. Thanks! Is your Q8_0 file the same as the one hosted directly on the Qwen GGUF page?
- danielhanchen 8mo agoNice! Yes Q8_0 is similar - the others are different since they use a calibration dataset.
- MrDrMcCoy 8mo agoStill hoping IQuest-Coder gets the same treatment :)
- bytesandbits 8mo agohow are you so fast man