4 ms·
ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/doc
by Maxious 7mo ago
ICYMI unsloth has had some major breakthroughs today with the Qwen3.5 local models https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
With the Qwen3.5 35B A3B at Q4 I've got 200k context running at 62.98 tokens per second on a local RTX5080 16GB.
- jychang 7mo agoNot really breakthroughs, more like bugfixes for their broken first batch.
- danielhanchen 7mo agoNo this is false - unsure if you saw our new blog - https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks which shows SOTA on nearly all bits, and we shared all our research as well
- jychang 7mo agoYeah, I saw that yesterday. The blog post does not explain why/how the Qwen 3.5 quants uploaded on 2/27 are different from the files uploaded on 2/24. Old 2/24 Q4_K_XL commit (pre bugfix files): https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/commit/7a8e0b23fcaf1a052ad02eb73f1c0627177e8325 https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/commit/7... Questions for a postmortem that the blog post left unanswered: - Why the change? Is it just to improve PPL/KLD? Sure, we can assume PPL and KLD are not perfect benchmarks. If yes, then why change the quantization anyways? Or was the old 2/24 quant actually much worse performing in the real world?I presume the Q4_K_XL quant using mxfp4 was the issue? If the 2/24 files having a lower PPL is an actual issue due to low quality tensors, then why not just say that? - What were the main tensors that had the quantizations changed from 2/24 to 2/27? Did you now quantize attention tensors differently? Or perhaps ssm? T - What was it changed from? Was it changed from mxfp4 or q4_k to q8, or something else? A quick sentence in the blog post saying "ok, we've confirmed that using mxfp4 (or q3 or whatever) in the attention/ssm/biases/norms/etc is a bad idea, we had that in our old models on 2/24 and our new models today are better" that would make it clear. As it's written, it's trying to both say "PPL/KLD don't actually reflect real world quality" and "we changed our quant to increase PPL/KLD" at the same time, which seems contradictory.
- zargon 7mo agoExplain what about that statement is false. Your original Q4_K_XL quant was broken. People noticing that it was a total outlier among other quants is what prompted this "research". Your own data proves that your new release fixes the bugs of your original, in order to match AesSedai's PPL. Fixing bugs is great. Searching for the best quant mix is helpful. I use your quants and appreciate your work. But whitewashing this situation dilutes trust and good will.
- Kayou 7mo agoWait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had
- segmondy 7mo agollama.cpp is designed for partial offloading, the most important part of the model will be loaded into the GPU and the rest on system ram. I run 500B+ models such as DeepSeek/KimiK2.5/GLM-5 without having that much GPU vram.
- pyuser583 7mo agoHow much do you use? I have lots of trouble figuring out what the limits are of a system with x amount of vram and y amounts of ram. How do you determine this?
- fc417fc802 7mo agoIdeally you'd have (parameter count) * (bits per parameter) VRAM for the entire (presumably quantized, don't forget to account for that) model. So very approximately 16 GiB for a 34B model quantized to 4 bits per parameter. You can spill to RAM in which case you at least want enough for a single active expert but really that's going to tank performance. If you're only "a bit" short of the full model the difference might not be all that large. These things are memory bandwidth limited so if you check out RAM, VRAM, and PCIe bandwidth what I wrote above should make sense. Also you should just ask your friendly local LLM these sorts of questions.
- pyuser583 7mo agoI usually do ask the llm what parameters to use. But that’s why I know so little about parameters!
- Maxious 7mo agoYep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe https://huggingface.co/blog/moe There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/ https://bknyaz.github.io/blog/2026/moe/
- mirekrusin 7mo ago2x RTX 4090, Q8, 256k context, 110 t/s
- instagib 7mo ago1 4090, Qwen3.5-35B-A3B-UD-MXFP4_MOE, 64k context, 122 t/s. Llama.cpp
- mirekrusin 7mo agoI believe it's mentioned that MXFP4 performs surprisingly bad, you may want to try other Q4s.
- danielhanchen 7mo agoOh I didn't expect this to be on HN haha - but yes for our new benchmarks for Qwen3.5, we devised a slightly different approach for quantization which we plan to roll out to all new models from now on!
- nnx 7mo agoCan you describe what is this slightly different approach and why it should work on all models?
- hedora 7mo agoNice! Your stuff ran LLMs extremely well on < $500 boxes (24-32GB ram) with iGPUS before this update. I’m eager to try it out, especially if 16GB is viable now.
- gundmc 7mo agoThe 5080 is 16GB VRAM, not system memory. I don't think you can get 24-32GB VRAM in a $500 box
- RS-232 7mo agoThat’s intriguing. I have the same card, maybe I should give it a go. Curious about your CPU/RAM/storage capacity as well. Any resources for configuring the local setup? My entire home media stack is a single compose file in a WSL distro so it would be cool if local LLM worked the same way.
- roxolotl 7mo agoWhat method are you using to do that? I’ve been playing with llama.cpp a lot lately and trying to figure out the cleanest options for getting a solid context window on 32gb vram and 64gb system ram.
- jychang 7mo ago32GB vram is more than enough for Qwen 3.5 35b You can just load the Q4_K_XL model like normal, and put all tensors on GPU without any -ot or --cpu-moe flags. If you need a massive context for some reason where model+kv cache won't fit in 32gb, then use -ot to move the ffn moe experts for 1-2 layers into RAM. You'll get a speed hit (due to loading params from slower RAM instead of fast VRAM) but it'll work.
- roxolotl 7mo agoNice ok I’ll play with that. I’m mostly just learning what’s possible. Qwen 3.5 35b has been great without any customizations but it’s interesting to learn what the options are.
- cpburns2009 7mo agoDoes llama.cpp support Qwen3.5 yet? When I tried it before, it failed saying "qwen35moe" is an unsupported architecture.
- reactordev 7mo agoYou would need the Dynamic 2.0 GGUF as discussed in the article. But mmmmmm, Q8_K_XL looks mighty nice.
- hnfong 7mo agoYes, but make sure you grab the latest llama.cpp release New model archs usually involve code changes.
- cpburns2009 7mo agoAwesome! It looks like the llama.cpp-hip AUR was updated today to b8179, and it works.
- sowbug 7mo agoIf you're running Ollama, you'll have to wait a little longer for its embedded version of llama.cpp to catch up. It can be a couple days or weeks behind.