3 ms·
Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on
by kentonv 2mo ago
Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard.
I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!
Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.
(Frontier models in comparison will write the app in one shot with no errors at all.)
Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart...
[0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!
- ChadNauseam 2mo agoI've been pretty impressed with Laguna. I downloaded their coding agent and have used it for a task here and there (the larger variant). Obviously it's nothing like a frontier LLM, but it surprised me with how good it was. And I think the model personality and way it talks is pretty pleasant
- trouve_search 2mo agoLaguna XS is MoE, however.
- kentonv 2mo agoOh! I think I missed that and assumed it wasn't because it performed similarly to the dense models. Interesting!
- SwellJoe 2mo agoAs an aside, the bigger S 2.1 runs faster than the dense Qwen 3.6 and Gemma 4 models on the same hardware (assuming the same hardware is big enough to run it) and definitely feels smarter, and more capable of long tasks, but it doesn't feel as heavily tuned for code as Qwen 3.6.
- sroussey 2mo agoWhat’s good for data extraction? I have a hard time getting models to just pull names and titles from a blob of text.
- rufasterisco 2mo agoBuild/vibe-code your own benchmark and find out, the work pays off and lets you try many models. Also try fine-tuning small models and see how they perform.
- sroussey 2mo agoOh, I have, and even Opus falls flat. I feel like regex would be better. Oh... maybe i can have Opus create such a regex...
- usagisushi 2mo agoIf you haven't yet, you might want to try gemma-4-E4B-it-qat or gemma-4-12B-it-qat with Structured Outputs. My main use case is tagging photos and generating headlines.
- dd8601fn 2mo agoI’m confused about the naming suggestion. Seems like the AxB bit differentiates pretty clearly, no?
- kentonv 2mo agoOnly if you know what it means. To many people these suffixes are gibberish. Also ollama doesn't actually show that suffix in its model index. Don't say "don't use ollama", you sound like a Linux user telling people not to use Mac. (I say this as a Linux user who despises Apple products myself.)
- zargon 2mo ago> ollama's model index typically only distinguishes variants by their size Don't use ollama. The entire project is just a series of stupid decisions like this.
- kristjansson 2mo agoHow they have any credibility after they told everyone they could run Deepseek R1 on their laptop (by giving the Qwen 7B distill the `deepseek` moniker)...
- Foobar8568 2mo agoAnd last time I have checked, you still can't run rerankers with it, yet you can download the models. See issue 3368, 2years old now.
- SirLordBoss 2mo agoWhat's the sota? The classic vLLM/llama.cpp? LM Studio? Unsloth studio any good?
- zargon 2mo agoAs a replacement for ollama, LM Studio is probably the most appropriate. Though I haven't tried Unsloth Studio yet. You get more control using llama.cpp directly, but there's more of a learning curve. vLLM has the best performance if you can fit your entire model into VRAM. llama.cpp is the usual go-to if you're partially loading into RAM. LM Studio is a sensible front-end to llama.cpp. ik_llama.cpp has more advanced CPU quantization strategies than llama.cpp. If you're running super large models mostly from RAM, ktransformers can sometimes be the highest performer, if it works for your model.
- literalAardvark 2mo agoOllama uses "consumer UI guidelines", it's the second worst AI tool to use for anything more than writing fanfic.
- msdz 2mo agoNow I have to ask – which is the worst?
- literalAardvark 2mo agoI'd have to go with whatever's embedded in Google searches
- woadwarrior01 2mo agoGood luck writing fanfics with primitive top-p and top-k samplers.
- literalAardvark 2mo agoBetter than average at fanfic really
- natrys 2mo agoThere is an old rule of thumb that says the quality of an MoE is equivalent to a dense model with the geometric mean of its total and active parameters. So, for example, the Qwen3.6 would be equivalent to a dense model with approximately sqrt(35×3) ≈ 10.25B parameters. Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).