4 ms·
I’m most excited about Qwen-30B-A3B. Seems like a good choice for offline/local-only coding assistants. Until now I found that open weight models were either n
by dcastm 1y ago
I’m most excited about Qwen-30B-A3B. Seems like a good choice for offline/local-only coding assistants.
Until now I found that open weight models were either not as good as their proprietary counterparts or too slow to run locally. This looks like a good balance.
- esafak 1y agoCould this variant be run on a CPU?
- moconnor 1y agoProbably very well
- htsh 1y agocurious, why the 30b MoE over the 32b dense for local coding? I do not know much about the benchmarks but the two coding ones look similar.
- Casteil 1y agoThe MoE version with 3b active parameters will run significantly faster (tokens/second) on the same hardware, by about an order of magnitude (i.e. ~4t/s vs ~40t/s)
- genpfault 1y ago> The MoE version with 3b active parameters ~34 tok/s on a Radeon RX 7900 XTX under today's Debian 13.
- tgtweak 1y agoAnd vmem use?
- genpfault 1y ago~18.6 GiB, according to nvtop. ollama 0.6.6 invoked with: # server OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve # client ollama run --verbose qwen3:30b-a3b ~19.8 GiB with: /set parameter num_ctx 32768
- tgtweak 1y agoVery nice, should run nicely on a 3090 as well. TY for this. update: wow, it's quite fast - 70-80t/s on LM Studio with a few other applications using GPU.
- kristianp 1y agoIt would be interesting to try, but for the Aider benchmark, the dense 32B model scores 50.2 and the 30B-A3B doesn't publish the Aider benchmark, so it may be poor.
- estsauver 1y agoIs that Qwen 2.5 or Qwen 3? I don't see a qwen 3 on the aider benchmark here yet: https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- aitchnyu 1y agoAs a human who asks AI to edit upto 50 SLOC at a time, is there value in models which score less than 50%? Im using the `gemini-2.0-flash-001` though.
- manmal 1y agoThe aider score mentioned in GP was published by Alibaba themselves, and is not yet on aider's leaderboard. The aider team will probably do their own tests and maybe come up with a different score.