4 ms·

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

TomBombadildoze 1mo ago
Beats Opus 4.6 at what exactly? It certainly isn't code. I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
- brandon272 1mo ago
  What configuration? What harness? These matter greatly to how a local model performs, in my experience.
- kennywinker 1mo ago
  https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-opus-4-6-adaptive#artificial-analysis-intelligence-index https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o... Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index https://artificialanalysis.ai/evaluations/artificial-analysi... Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4. If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.
fshr 1mo ago
> 5060 I think you're referring to a 5060Ti 16GB, yes? 32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.
- kennywinker 1mo ago
  Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall. But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.

Beats Opus 4.6 at what exactly? It certainly isn't code. I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.

brandon272 1mo ago
What configuration? What harness? These matter greatly to how a local model performs, in my experience.
kennywinker 1mo ago
https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-opus-4-6-adaptive#artificial-analysis-intelligence-index https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o... Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index https://artificialanalysis.ai/evaluations/artificial-analysi... Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4. If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.

fshr 1mo ago

> 5060 I think you're referring to a 5060Ti 16GB, yes? 32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.

kennywinker 1mo ago
Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall. But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.