4 ms·
What sort of hardware will run Qwen3-Coder-480B-A35B-Instruct? With the performance apparently comparable to Sonnet some of the heavy Claude Code users could b
by jasonthorsness 1y ago
What sort of hardware will run Qwen3-Coder-480B-A35B-Instruct?
With the performance apparently comparable to Sonnet some of the heavy Claude Code users could be interested in running it locally. They have instructions for configuring it for use by Claude Code. Huge bills for usage are regularly shared on X, so maybe it could even be economical (like for a team of 6 or something sharing a local instance).
- sourcecodeplz 1y agoWith RAM you would need at least 500gb to load it but some 100-200gb more for context too. Pair it with a 24gb GPU and the speed will be 10t/s, at least, I estimate.
- danielhanchen 1y agoOh yes for the FP8, you will need 500GB ish. 4bit around 250GB - offloading MoE experts / layers to RAM will definitely help - as you mentioned a 24GB card should be enough!
- vFunct 1y agoDo we know if the full model is FP8 or FP16/BF16? The hugging face page says BF16: https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct So likely it needs 2x the memory.
- danielhanchen 1y agoI think it's BF16 trained then quantized to FP8, but unsure fully - I was also trying to find out if they used FP8 for training natively!
- jychang 1y agoQwen uses 16bit, Kimi and Deepseek uses FP8.
- danielhanchen 1y agoOh ok cool thanks!
- danielhanchen 1y agoI'm currently trying to make dynamic GGUF quants for them! It should use 24GB of VRAM + 128GB of RAM for dynamic 2bit or so - they should be up in an hour or so: https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruc.... On running them locally, I do have docs as well: https://docs.unsloth.ai/basics/qwen3-coder https://docs.unsloth.ai/basics/qwen3-coder
- gardnr 1y agoLegend
- danielhanchen 1y ago:)
- zettabomb 1y agoAny significant benefits at 3 or 4 bit? I have access to twice that much VRAM and system RAM but of course that could potentially be better used for KV cache.
- sourcecodeplz 1y agoFor coding you want more precision so the higher the quant the better. But there is discussion if a smaller model in higher quant is better than a larger one in lower quant. Need to test for yourself with your use cases I'm afraid. e: They did announce smaller variants will be released.
- danielhanchen 1y agoYes the higher the quant, the better! The other approach is dynamically choosing to upcast some layers!
- segmondy 1y agoI can say that this really works great, I'm a heavy user of the unsloth dyanmic quants. I run DeepSeek v3/r1 in Q3, and ernie-300b and KimiK2 in Q3 too. Amazing performance. I run Qwen3-235b in both Q4 and Q8 and can barely tell the difference so much so that I just keep Q4 since it's twice as fast.
- ilaksh 1y agoTo run the real version with the bench arks they give, it would be a nonquantized non distilled version. So I am guessing that is a cluster of 8 H200s if you want to be more or less up to date. They have B200s now which are much faster but also much more expensive. $300,000+ You will see people making quantized distilled versions but they never give benchmark results.
- danielhanchen 1y agoOh you can run the Q8_0 / Q8_K_XL which is nearly equivalent to FP8 (maybe off by 0.01% or less) -> you will need 500GB of VRAM + RAM + Disk space. Via MoE layer offloading, it should function ok
- btian 1y agoDo need to be super fancy. Just RTX Pro 6000 and 256GB of RAM.
- simonw 1y agoThere's a 4bit version here that uses around 272GB of RAM on a 512GB M3 Mac Studio: https://huggingface.co/mlx-community/Qwen3-Coder-480B-A35B-Instruct-4bit https://huggingface.co/mlx-community/Qwen3-Coder-480B-A35B-I... - see video: https://x.com/awnihannun/status/1947771502058672219 https://x.com/awnihannun/status/1947771502058672219 That machine will set you back around $10,000.
- jychang 1y agoYou can get similar performance on an Azure HX vm: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/high-performance-compute/hx-series?tabs=sizebasic https://learn.microsoft.com/en-us/azure/virtual-machines/siz...
- osti 1y agoHow? These don't even have GPU's right?
- jychang 1y agoThey have similar memory bandwidth compared to the Mac Studio. You can run it off CPU at the same speed.
- osti 1y agoInteresting, so with enough memory bandwidth, even the server CPU has enough compute to do inference on a rather large model? Enough to compete against M4 gpu? Edit: I just aked chatgpt and it says with no memory bandwidth bottleneck, i can still only achieve around 1 token/s from a 96 core cpu.
- timschmidt 1y agoFor a single user prompting with one or few prompts at a time, compute is not the bottleneck. Memory bandwidth is. This is because the entire model's weights must be run through the algorithm many times per prompt. This is also why multiplexing many prompts at the same time is relatively easy and effective, as many matrix multiplications can happen in the time it takes to do a single fetch from memory.
- Avlin67 1y agoxeon 6980P which now costs 6K€ instead of 17K€
- 827a 1y agoThe initial set of prices on OpenRouter look pretty similar to Claude Sonnet 4, sadly.
- chisleu 1y agoA mac studio can run it at 4bit. Maybe at 6 bit.
- chisleu 1y agoA Mac Studio 512GB can run it in 4bit quantization. I'm excited to see unsloth dynamic quants for this today.