3 ms·
This is exactly the kind of model that's been needed in the middle. Realistically self-hosted, Good Enough intelligence, MoE so it's fast on limited bandwidth s
by SwellJoe 3mo ago
This is exactly the kind of model that's been needed in the middle. Realistically self-hosted, Good Enough intelligence, MoE so it's fast on limited bandwidth systems like Strix Halo and DGX Spark.
For a while there's been nothing to run on my Strix Halo that's notably better than what I can run on my dual 32GB GPU desktop (Gemma 4 or Qwen 3.6 dense models), but this seems likely to be the step up in size that actually works better than those.
- mandelken 3mo agoOn Strix Halo, have a look at the work antirez is doing with dwarfstar (antirez/ds4) for deepseek v4 flash at reasonable speeds and quality.
- SwellJoe 3mo agoI've tried it, but, it's not reasonable speed, at all. It's 9-13 tokens per second, which is not usable interactively and not worth using for long-running API stuff when DeepSeek V4 Pro is so cheap via their API. Laguna S 2.1 runs at 15-28 tokens per second, depending on context and...something about how long it's been running, which is very comfortable for chatting, but still not usable for interactive agentic coding. Their `pool` agent just times out when I try to use it with the Strix Halo-hosted instance of the model.