3 ms·

Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.

by simonw 2mo ago

SwellJoe 2mo ago
Yes, default everything, no tuning, 8_K_XL Unsloth quantization on dual Radeon V620 GPUs (which aren't blazing, but faster than the Strix Halo).
- syntaxing 2mo ago
  What speed do you get on this setup? Im tempted to use the same GPU.
  - SwellJoe 2mo ago
    It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting. If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
    - SwellJoe 2mo ago
      Update on this: When I enable tensor parallelism in llama.cpp, I see 25-33 t/s. With reasoning effort set to medium, Qwen 3.8 finished the same task that previously took 11 hours in a little over three hours, which is still more than three times what most of the large models required including Opus 4.8, and nine times what GPT 5.5 (the fastest of the models I've used) needed for a similar task. So, it's still not fast enough for comfort, but it's much faster than the first run. And, I guess, faster than writing the code myself.
      - syntaxing 2mo ago
        Thanks for the update! I wonder if a Blackwell GPU would be noticeably faster. Which vendor did you end up using? I want to get a gigabyte one but thats be OOS for months.
        SwellJoe 2mo ago
        The Blackwell and Strix Halo will be similar to each other and much slower than the number I'm getting on the dual V620 setup (I see about 10-15 t/s on my Strix Halo with this model at 8-bit quantization depending on context). Prefill is generally quite a bit faster on the DGX Spark and token generation slightly faster on the Strix Halo, as I understand it. But, there are better software efficiency improvements for the Spark line. This model is far from usable on current AMD or Nvidia 128GB AI machines, IMHO, they just don't have the memory bandwidth, especially since it chews so many tokens for any task. If you want to run this specific model, two (or more) 32GB GPUs with decent memory bandwidth is the right way to do it. It doesn't benefit from the larger memory of the Strix Halo. There's enough room for full context and 8-bit quantized model in 64GB. But, it's really a terrible time to buy hardware. MoE models are a much better fir for the Spark and Strix Halo; you can run Laguna S2.1 (slowly) or one of the Qwen 3.6 MoE fine-tunes (pretty quick). Ling 3.0 Flash also looks promising. Nemotron 3.5 Lightning in the MXFP4 quantization absolutely flies on the Strix Halo at 65-80 t/s, but it's dumb. But, all of those are weaker than Qwen 3.8 27B for coding.
dofm 2mo ago
You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it. The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode. It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough. Really useful, thanks.
- anon373839 2mo ago
  > Qwen 3.6 35B A3B on medium thinking mode Qwen 3.6 doesn’t have configurable reasoning effort, does it?
  - dofm 2mo ago
    Hm — brain jumped tracks a bit there at nearly 4am. I'm talking about budget — I mean limiting it to 2048 tokens. … for one or other of the models I tested at the same time, in llama-server, there is a dropdown that offered options (unlimited, max, medium which was 2048) (I've tested so many of these things now that they are beginning to blur.) I thought that was llama-server with Qwen 35B, just checked and it's not. Nor is it Gemma 4 26B. Perhaps it was Ternary Bonsai which I tested again and deleted earlier. Anyway I took to clipping Qwen 3.6 35B at 2048 tokens reasoning in LM Studio and elsewhere, and it did OK at that (because it often loops like mad on an ambiguous prompt if not curtailed). FWIW I just rechecked outputs and I am a bit over-optimistic when I say 3.8 27B 's non-thinking output is that good. I spotted a couple of subtle errors in my tests that Low thinking didn't fail on. It is good, but it is not quite Qwen 3.6 35B thinking level.

Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.

by simonw 2mo ago

Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.

SwellJoe 2mo ago

Yes, default everything, no tuning, 8_K_XL Unsloth quantization on dual Radeon V620 GPUs (which aren't blazing, but faster than the Strix Halo).

syntaxing 2mo ago
What speed do you get on this setup? Im tempted to use the same GPU.
- SwellJoe 2mo ago
  It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting. If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
  - SwellJoe 2mo ago
    Update on this: When I enable tensor parallelism in llama.cpp, I see 25-33 t/s. With reasoning effort set to medium, Qwen 3.8 finished the same task that previously took 11 hours in a little over three hours, which is still more than three times what most of the large models required including Opus 4.8, and nine times what GPT 5.5 (the fastest of the models I've used) needed for a similar task. So, it's still not fast enough for comfort, but it's much faster than the first run. And, I guess, faster than writing the code myself.
    - syntaxing 2mo ago
      Thanks for the update! I wonder if a Blackwell GPU would be noticeably faster. Which vendor did you end up using? I want to get a gigabyte one but thats be OOS for months.
      - SwellJoe 2mo ago
        The Blackwell and Strix Halo will be similar to each other and much slower than the number I'm getting on the dual V620 setup (I see about 10-15 t/s on my Strix Halo with this model at 8-bit quantization depending on context). Prefill is generally quite a bit faster on the DGX Spark and token generation slightly faster on the Strix Halo, as I understand it. But, there are better software efficiency improvements for the Spark line. This model is far from usable on current AMD or Nvidia 128GB AI machines, IMHO, they just don't have the memory bandwidth, especially since it chews so many tokens for any task. If you want to run this specific model, two (or more) 32GB GPUs with decent memory bandwidth is the right way to do it. It doesn't benefit from the larger memory of the Strix Halo. There's enough room for full context and 8-bit quantized model in 64GB. But, it's really a terrible time to buy hardware. MoE models are a much better fir for the Spark and Strix Halo; you can run Laguna S2.1 (slowly) or one of the Qwen 3.6 MoE fine-tunes (pretty quick). Ling 3.0 Flash also looks promising. Nemotron 3.5 Lightning in the MXFP4 quantization absolutely flies on the Strix Halo at 65-80 t/s, but it's dumb. But, all of those are weaker than Qwen 3.8 27B for coding.

dofm 2mo ago

You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it. The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode. It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough. Really useful, thanks.

anon373839 2mo ago
> Qwen 3.6 35B A3B on medium thinking mode Qwen 3.6 doesn’t have configurable reasoning effort, does it?
- dofm 2mo ago
  Hm — brain jumped tracks a bit there at nearly 4am. I'm talking about budget — I mean limiting it to 2048 tokens. … for one or other of the models I tested at the same time, in llama-server, there is a dropdown that offered options (unlimited, max, medium which was 2048) (I've tested so many of these things now that they are beginning to blur.) I thought that was llama-server with Qwen 35B, just checked and it's not. Nor is it Gemma 4 26B. Perhaps it was Ternary Bonsai which I tested again and deleted earlier. Anyway I took to clipping Qwen 3.6 35B at 2048 tokens reasoning in LM Studio and elsewhere, and it did OK at that (because it often loops like mad on an ambiguous prompt if not curtailed). FWIW I just rechecked outputs and I am a bit over-optimistic when I say 3.8 27B 's non-thinking output is that good. I spotted a couple of subtle errors in my tests that Low thinking didn't fail on. It is good, but it is not quite Qwen 3.6 35B thinking level.