3 ms·
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting
by peri-cl 2mo ago
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.
- expedited123 2mo agoMind sharing your laptops specs? Just interested to see what is needed to locally run Qwen3.6-35B-A3B
- peri-cl 2mo agoDon't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4)) I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting). Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
- expedited123 2mo agoThanks! I don't have any knowledge of running models locally. I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
- seanmcdirmid 2mo agonot parent, but 4-bit quantization is generally consider a good trade off for speed/performance, so you might use it even when you aren't on consumer hardware, but definitely when you are on consumer hardware.
- cyanydeez 2mo agoyeah, that's the A3B part; going up to A5B would probably also feel comfortable. on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.
- peri-cl 2mo ago[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s. MoE is literally 5x times faster (on CPU) than comparable dense Qwen! Tested: Qwen3.6-35B-A3B-UD-Q4_K_XL Qwen3.8-27B-UD-Q4_K_XL
- qeternity 2mo ago35A3 might be more comparable to 10 dense. 27 dense is far more capable than 35A3.
- kzrdude 2mo agoSo you happen to know how to demonstrate this higher capacity, what do we mean by that? Can understand bigger codebases? And how do we explain it, what part of the model does it?
- searealist 2mo agoIf you are not already using MTP, you should be able to get ~2x decode tokens/s with Qwen 3.8 27B.
- rasengan0 2mo agoGot Qwen3.8 to run on my Framework 12 Intel Core 13 Gen Raptor Lake i5-1334U small laptop with 48G RAM stick: llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL but it failed my basic prompt to compose a vim regex to match CamelCaseWords downgraded a bit with Q4_K_M from ollama run qwen3.8:27b and /set nothink and at least 1 regex matched FooBar prompt eval at 2.8 t/s eval at 0.94t/s I particularly enjoyed this usage of the regex: /%\1\%/ ... onward for 700+ characters of \%\/ :-)