6 ms·

I'm currently making 2bit to 8bit GGUFs for local deployment! Will be up in an hour or so at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF

I'm currently making 2bit to 8bit GGUFs for local deployment! Will be up in an hour or so at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruc... Also docs on running it in a 24GB GPU + 128 to 256GB of RAM here: https://docs.unsloth.ai/basics/qwen3-coder https://docs.unsloth.ai/basics/qwen3-coder

mathrawka 1y ago
Looks like the docs have a typo: Recommended context: 65,536 tokens (can be increased) That should be recommended token output, as shown in the official docs as: Adequate Output Length: We recommend using an output length of 65,536 tokens for most queries, which is adequate for instruct models.
- danielhanchen 1y ago
  Oh thanks - so the output can be any length you like - I'm actually also making 1 million context length GGUFs as well! https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-GGUF https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruc...
sgammon 1y ago
Cool, thanks! I'd like to try it
- danielhanchen 1y ago
  It just got uploaded! I made some docs as well on how to run it at https://docs.unsloth.ai/basics/qwen3-coder https://docs.unsloth.ai/basics/qwen3-coder
Jayakumark 1y ago
What will be the approx token/s prompt processing and generation speed with this setup on RTX 4090?
- danielhanchen 1y ago
  I also just made IQ1_M which needs 160GB! If you have 160-24 = 136 ish of RAM as well, then you should get 3 tokens to 5 ish per second. If you don't have enough RAM, then < 1 token / s
jdright 1y ago
Any idea if there is a way to run on 256gb ram + 16gb vram with usable performance, even if barely?
- danielhanchen 1y ago
  Yes! 3bit maybe 4bit can also fit! llama.cpp has MoE offloading so your GPU holds the active experts and non MoE layers, thus you only need 16GB to 24GB of VRAM! I wrote about how to do in this section: https://docs.unsloth.ai/basics/qwen3-coder#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-coder#improving-generat...
  - jdright 1y ago
    awesome documentation, I'll try this. thank you!
Rockytu1 1y ago
[dead]
andai 1y ago
I've been reading about your dynamic quants, very cool. Does your library let me produce these, or only run them? I'm new to this stuff.
- danielhanchen 1y ago
  Thank you! Oh currently not sadly - we might publish some stuff on it in the future!
Abishek_Muthian 1y ago
Thank you for your work, does the Qwen3-Coder offer significant advantage over Qwen2.5-coder for non-agentic tasks like just plain autocomplete and chat?
- danielhanchen 1y ago
  Oh it should be better, especially since the model was specifically designed for coding tasks! You can disable the tool calling parts of the model!
appleaday1 1y ago
hello sir
- danielhanchen 1y ago
  hi!
gnulinux 1y ago
Do 2bit quantizations really work? All the ones I've seen/tried were completely broken even when 4bit+ quantizations worked perfectly. Even if it works for these extremely large models, is it really much better than using something slightly smaller on 4 or 5 bit quant?
- danielhanchen 1y ago
  Oh the Unsloth dynamic ones are not 2bit at all - it's a mixture of 2, 3, 4, 5, 6 and sometimes 8bit. Important layers are in 8bit, 6bit. Less important ones are left in 2bit! I talk more about it here: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs
  - CMCDragonkai 1y ago
    How do you decide which layers are the important ones?
    - danielhanchen 1y ago
      I wrote approximately in the blog about it and linked some papers! I also wrote about it here - https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit - one has to inspect the activation and weight quantization errors!
      - blensor 1y ago
        So you are basically looking at "fMRI" of the "brain" while it's doing a wide range of things and cutting out the things that stay dark the most?
        danielhanchen 1y ago
        Oh that's a good analogy! Yes that sounds right!
    - 1y ago
PeterStuer 1y ago
I had given up long time ago on self hosted transformer models for coding because the SOTA was definetly in favor of SaaS. This might just give me another try. Would llama.cpp support multiple (rtx 3090, no nvlink hw bridge) GPUs over PCIe4? (Rest of the machine is 32 CPU cores, 256GB RAM)
- danielhanchen 1y ago
  Oh yes llama.cpp's trick is it supports any hardware setup! It might be a bit slower, but it should function well!
- jychang 1y ago
  How fast you run this model will strongly depend on if you have DDR4 or DDR5 ram. You will be mostly using 1 of your 3090s. The other one will be basically doing nothing. You CAN put the MoE weights on the 2nd 3090, but it's not going to speed up inference much, like <5% speedup. As in, if you lack a GPU, you'd be looking at <1 token/sec speeds depending on how fast your CPU does flops, and if you have a single 3090 you'd be doing 10tokens/ec, but with 2 3090s you'll still just be doing maybe 11tok/sec. These numbers are made up, but you get the idea. Qwen3 Coder 480B is 261GB for IQ4_XS, 276GB for Q4_K_XL, so you'll be putting all the expert weights in RAM. That's why your RAM bandwidth is your limiting factor. I hope you're running off a workstation with dual cpus and 12 sticks of DDR5 RAM per CPU, which allows you to have 24 channel DDR5 RAM.
  - PeterStuer 1y ago
    1 CPU, DDR4 ram
    - jychang 1y ago
      How many channels of DDR4 ram? What speed is it running at? DDR4-3200? The (approximate) equation for milliseconds per token, is: Time for token generation = (number of params active in the model)*(quantization size in bits)/8 bits*[(percent of active params in common weights)/(memory bandwidth of GPU) + (percent of active params in experts)/(memory bandwidth of system RAM)]. This equation ignores prefill (prompt processing) time. This assumes the CPU and GPU is fast enough compute-wise to do the math, and the bottleneck is memory bandwidth (this is usually true). So for example, if you are running Kimi K2 (32b active params per token, 74% of those params are experts, 26% of those params are common params/shared expert) at Q4 quantization (4 bits per param), and have a 3090 gpu (935GB/sec) and an AMD Epyc 9005 cpu with 12 channel DDR5-6400 (614GB/sec memory bandwidth), then: Time for token generation = (32b params)*(4bits/param)/8 bits*[(26%)/(935 GB/s) + (74%)/(614GB/sec)] = 23.73 ms/token or ~42 tokens/sec. https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B%2826%25%29%2F%28935+GB%2Fsec%29+%2B+%2874%25%29%2F%28614GB%2Fsec%29%5D%29 https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B... Notice how this equation explains how the second 3090 is pretty much useless. If you load up the common weights on the first 3090 (which is standard procedure), then the 2nd 3090 is just "fast memory" for some expert weights. If the quantized model is 256GB (rough estimate, I don't know the model size off the top of my head), and common weights are 11GB (this is true for Kimi K2, I don't know if it's true for Qwen3, but this is a decent rough estimate), then you have 245GB of "expert" weights. Yes, this is generally the correct ratio for MoE models, Deepseek R1 included. If you put 24GB of that 245GB on your second 3090, you have 935GB/sec speed on... 24/245 or ~10% of each token. In my Kimi K2 example above, you start off with 18.08ms per token spent reading the model from RAM, so even if your 24GB on your GPU was infinitely fast, it would still take... about 16ms per token reading from RAM. Or in total about 22ms/token, or in total 45 tokens/sec. That's with an infinitely fast 2nd GPU, you get a speedup of merely 3 tokens/sec.
clownpenis_fart 1y ago
[dead]
colordrops 1y ago
Would a version of this ever be possible to run on a machine with a 16GB gpu and 64gb RAM?
ashvardanian 1y ago
Thanks for the uploads! Was reading through the Unsloth docs for Qwen3-Coder before I found the HN thread :) What would be a reasonable throughput level to expect from running 8-bit or 16-bit versions on 8x H200 DGX systems?
- danielhanchen 1y ago
  Oh 8*H200 is nice - for llama.cpp definitely look at https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locall... - llama.cpp has a high throughput mode which should be helpful. You should be able to get 40 to 50 tokens / s in the minimum. High throughput mode + a small draft model might get you 100 tokens / s generation

mathrawka 1y ago

Looks like the docs have a typo: Recommended context: 65,536 tokens (can be increased) That should be recommended token output, as shown in the official docs as: Adequate Output Length: We recommend using an output length of 65,536 tokens for most queries, which is adequate for instruct models.

danielhanchen 1y ago
Oh thanks - so the output can be any length you like - I'm actually also making 1 million context length GGUFs as well! https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-1M-GGUF https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruc...

sgammon 1y ago

Cool, thanks! I'd like to try it

danielhanchen 1y ago
It just got uploaded! I made some docs as well on how to run it at https://docs.unsloth.ai/basics/qwen3-coder https://docs.unsloth.ai/basics/qwen3-coder

Jayakumark 1y ago

What will be the approx token/s prompt processing and generation speed with this setup on RTX 4090?

danielhanchen 1y ago
I also just made IQ1_M which needs 160GB! If you have 160-24 = 136 ish of RAM as well, then you should get 3 tokens to 5 ish per second. If you don't have enough RAM, then < 1 token / s

jdright 1y ago

Any idea if there is a way to run on 256gb ram + 16gb vram with usable performance, even if barely?

danielhanchen 1y ago
Yes! 3bit maybe 4bit can also fit! llama.cpp has MoE offloading so your GPU holds the active experts and non MoE layers, thus you only need 16GB to 24GB of VRAM! I wrote about how to do in this section: https://docs.unsloth.ai/basics/qwen3-coder#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-coder#improving-generat...
- jdright 1y ago
  awesome documentation, I'll try this. thank you!

Rockytu1 1y ago

[dead]

andai 1y ago

I've been reading about your dynamic quants, very cool. Does your library let me produce these, or only run them? I'm new to this stuff.

danielhanchen 1y ago
Thank you! Oh currently not sadly - we might publish some stuff on it in the future!

Abishek_Muthian 1y ago

Thank you for your work, does the Qwen3-Coder offer significant advantage over Qwen2.5-coder for non-agentic tasks like just plain autocomplete and chat?

danielhanchen 1y ago
Oh it should be better, especially since the model was specifically designed for coding tasks! You can disable the tool calling parts of the model!

appleaday1 1y ago

hello sir

danielhanchen 1y ago
hi!

gnulinux 1y ago

Do 2bit quantizations really work? All the ones I've seen/tried were completely broken even when 4bit+ quantizations worked perfectly. Even if it works for these extremely large models, is it really much better than using something slightly smaller on 4 or 5 bit quant?

danielhanchen 1y ago
Oh the Unsloth dynamic ones are not 2bit at all - it's a mixture of 2, 3, 4, 5, 6 and sometimes 8bit. Important layers are in 8bit, 6bit. Less important ones are left in 2bit! I talk more about it here: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs
- CMCDragonkai 1y ago
  How do you decide which layers are the important ones?
  - danielhanchen 1y ago
    I wrote approximately in the blog about it and linked some papers! I also wrote about it here - https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit - one has to inspect the activation and weight quantization errors!
    - blensor 1y ago
      So you are basically looking at "fMRI" of the "brain" while it's doing a wide range of things and cutting out the things that stay dark the most?
      - danielhanchen 1y ago
        Oh that's a good analogy! Yes that sounds right!
  - 1y ago

PeterStuer 1y ago

I had given up long time ago on self hosted transformer models for coding because the SOTA was definetly in favor of SaaS. This might just give me another try. Would llama.cpp support multiple (rtx 3090, no nvlink hw bridge) GPUs over PCIe4? (Rest of the machine is 32 CPU cores, 256GB RAM)

danielhanchen 1y ago
Oh yes llama.cpp's trick is it supports any hardware setup! It might be a bit slower, but it should function well!
jychang 1y ago
How fast you run this model will strongly depend on if you have DDR4 or DDR5 ram. You will be mostly using 1 of your 3090s. The other one will be basically doing nothing. You CAN put the MoE weights on the 2nd 3090, but it's not going to speed up inference much, like <5% speedup. As in, if you lack a GPU, you'd be looking at <1 token/sec speeds depending on how fast your CPU does flops, and if you have a single 3090 you'd be doing 10tokens/ec, but with 2 3090s you'll still just be doing maybe 11tok/sec. These numbers are made up, but you get the idea. Qwen3 Coder 480B is 261GB for IQ4_XS, 276GB for Q4_K_XL, so you'll be putting all the expert weights in RAM. That's why your RAM bandwidth is your limiting factor. I hope you're running off a workstation with dual cpus and 12 sticks of DDR5 RAM per CPU, which allows you to have 24 channel DDR5 RAM.
- PeterStuer 1y ago
  1 CPU, DDR4 ram
  - jychang 1y ago
    How many channels of DDR4 ram? What speed is it running at? DDR4-3200? The (approximate) equation for milliseconds per token, is: Time for token generation = (number of params active in the model)*(quantization size in bits)/8 bits*[(percent of active params in common weights)/(memory bandwidth of GPU) + (percent of active params in experts)/(memory bandwidth of system RAM)]. This equation ignores prefill (prompt processing) time. This assumes the CPU and GPU is fast enough compute-wise to do the math, and the bottleneck is memory bandwidth (this is usually true). So for example, if you are running Kimi K2 (32b active params per token, 74% of those params are experts, 26% of those params are common params/shared expert) at Q4 quantization (4 bits per param), and have a 3090 gpu (935GB/sec) and an AMD Epyc 9005 cpu with 12 channel DDR5-6400 (614GB/sec memory bandwidth), then: Time for token generation = (32b params)*(4bits/param)/8 bits*[(26%)/(935 GB/s) + (74%)/(614GB/sec)] = 23.73 ms/token or ~42 tokens/sec. https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B%2826%25%29%2F%28935+GB%2Fsec%29+%2B+%2874%25%29%2F%28614GB%2Fsec%29%5D%29 https://www.wolframalpha.com/input?i=1+sec+%2F+%2816GB+*+%5B... Notice how this equation explains how the second 3090 is pretty much useless. If you load up the common weights on the first 3090 (which is standard procedure), then the 2nd 3090 is just "fast memory" for some expert weights. If the quantized model is 256GB (rough estimate, I don't know the model size off the top of my head), and common weights are 11GB (this is true for Kimi K2, I don't know if it's true for Qwen3, but this is a decent rough estimate), then you have 245GB of "expert" weights. Yes, this is generally the correct ratio for MoE models, Deepseek R1 included. If you put 24GB of that 245GB on your second 3090, you have 935GB/sec speed on... 24/245 or ~10% of each token. In my Kimi K2 example above, you start off with 18.08ms per token spent reading the model from RAM, so even if your 24GB on your GPU was infinitely fast, it would still take... about 16ms per token reading from RAM. Or in total about 22ms/token, or in total 45 tokens/sec. That's with an infinitely fast 2nd GPU, you get a speedup of merely 3 tokens/sec.

clownpenis_fart 1y ago

[dead]

colordrops 1y ago

Would a version of this ever be possible to run on a machine with a 16GB gpu and 64gb RAM?

ashvardanian 1y ago

Thanks for the uploads! Was reading through the Unsloth docs for Qwen3-Coder before I found the HN thread :) What would be a reasonable throughput level to expect from running 8-bit or 16-bit versions on 8x H200 DGX systems?

danielhanchen 1y ago
Oh 8*H200 is nice - for llama.cpp definitely look at https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locall... - llama.cpp has a high throughput mode which should be helpful. You should be able to get 40 to 50 tokens / s in the minimum. High throughput mode + a small draft model might get you 100 tokens / s generation