6 ms·

Running 3.5 9B on my ASUS 5070ti 16G with lm studio gives a stable ~100 tok/s. This outperforms the majority of online llm services and the actual quality of ou

Running 3.5 9B on my ASUS 5070ti 16G with lm studio gives a stable ~100 tok/s. This outperforms the majority of online llm services and the actual quality of output matches the benchmark. This model is really something, first time ever having usable model on consumer-grade hardware.

yangikan 7mo ago
Do you point claude code to this? The orchestration seems to be very important.
- teaearlgraycold 7mo ago
  I loaded Qwen into LM Studio and then ran Oh My Pi. It automatically picked up the LM Studio API server. For some reason the 35B A3B model had issues with Oh My Pi's ability to pass a thinking parameter which caused it to crash. 27B did not have that issue for me but it's much slower. Here's how I got the 35B model to work: https://gist.github.com/danthedaniel/c1542c65469fb1caafabe1371e8ed445 https://gist.github.com/danthedaniel/c1542c65469fb1caafabe13... The 35B model is still pretty slow on my machine but it's cool to see it working.
- badgersnake 7mo ago
  I’ve tried it on Claude code, Found it to be fairly crap. It got stuck in a loop doing the wrong thing and would not be talked out of it. I’ve found this bug that would stop it compiling right after compiling it, that sort of thing. Also seemed to ignore fairly simple instructions in CLAUDE.md about building and running tests.
- tommyjepsen 7mo ago
  I ran the Qwen3 Coder 30B through LM Studio and with OpenCode(Instead of Claude code). Did decent on M4 Max 32GB. https://www.tommyjepsen.com/blog/run-llm-locally-for-coding https://www.tommyjepsen.com/blog/run-llm-locally-for-coding
- andsoitis 7mo ago
  I use Claude Code for agentic coding but it is better to use qwen3-coder in that case. It qwen3-coder is better for code generation and editing, strong at multi-file agentic tasks, and is purpose-built for coding workflows. In contrast, qwen3.5 is more capable at general reasoning, better at planning and architecture decisions, good balance of coding and thinking.
- Aurornis 7mo ago
  The 9B models are not useful for coding outside of very simple requests. Qwen3.5 is confusing a lot of newcomers because it is very confident in the answers it gives. It can also regurgitate solutions to common test requests like “make a flappy bird clone” which misleads users into thinking it’s genetically smart. Using the Qwen3.5 models for longer tasks and inspecting the output is a little more disappointing. They’re cool for something I can run locally but I don’t agree with all of the claims about being Sonnet-level quality (including previous Sonnet versions) in my experience with the larger models. The 9B model is not going to be close to Sonnet in any way.
lukan 7mo ago
What exact model are you using? I have a 16GB GPU as well, but have never run a local model so far. According to the table in the article, 9B and 8-bit -> 13 GB and 27B and 3-bit seem to fit inside the memory. Or is there more space required for context etc?
- vasquez 7mo ago
  It depends on the task, but you generally want some context. These models can do things like OCR and summarize a pdf for you, which takes a bit of working memory. Even more so for coding CLIs like opencode-ai, qwen code and mistral ai. Inference engines like llama.cpp will offload model and context to system ram for you, at the cost of performance. A MoE like 35B-A3B might serve you better than the ones mentioned, even if it doesn't fit entirely on the GPU. I suggest testing all three. Perhaps even 122-A10B if you have plenty of system ram. Q4 is a common baseline for simple tasks on local models. I like to step up to Q5/Q6 for anything involving tool use on the smallish models I can run (9B and 35B-A3B). Larger models tolerate lower quants better than small ones, 27B might be usable at 3 bpw where 9B or 4B wouldn't. You can also quantize the context. On llama.cpp you'd set the flags -fa on, -ctk x and ctv y. -h to see valid parameters. K is more sensitive to quantization than V, don't bother lowering it past q8_0. KV quantization is allegedly broken for Qwen 3.5 right now, but I can't tell.
throwdbaaway 7mo ago
There are Qwen3.5 27B quants in the range of 4 bits per weight, which fits into 16G of VRAM. The quality is comparable to Sonnet 4.0 from summer 2025. Inference speed is very good with ik_llama.cpp, and still decent with mainline llama.cpp.
- teaearlgraycold 7mo ago
  Qwen3.5 35B A3B is much much faster and fits if you get a 3 bit version. How fast are you getting 27B to run? On my M3 Air w/ 24GB of memory 27B is 2 tok/s but 35B A3B is 14-22 tok/s which is actually usable.
  - ece 7mo ago
    The 27B is rated slightly higher for SWE-bench.
  - throwdbaaway 7mo ago
    Using ik_llama.cpp to run a 27B 4bpw quant on a RTX 3090, I get 1312 tok/s PP and 40.7 tok/s TG at zero context, dropping to 1009 tok/s PP and 36.2 tok/s TG at 40960 context. 35B A3B is faster but didn't do too well in my limited testing.
    - ranger_danger 7mo ago
      with regular llama.cpp on a 3070ti I get 60tok/s TG with the 9B model, it's quite impressive.
  - ranger_danger 7mo ago
    Don't sleep on the 9B version either, I get much faster speeds and can't tell any difference in quality. On my 3070ti I get ~60tok/s with it, and half that with the 35B-A3B.
  - andai 7mo ago
    27B needs less memory and does better on benchmarks, but 35B-A3B seems to run roughly twice as fast.
smokel 7mo ago
> This outperforms the majority of online llm services I assume you mean outperforms in speed on the same model, not in usability compared to other more capable models. (For those who are getting their hopes up on using local LLMs to be any replacement for Sonnet or Opus.)
- moffkalast 7mo ago
  Obviously it's not going to be of a paid tier 2T sized SOTA model quality, but it can probably roughly match Haiku at the very least. And for tasks that aren't super complex that's already enough. Personally though, I find Qwen useless for anything but coding tasks because if its insufferable sycophancy. It's like 4o dialed up to 20, every reply starts with "You are absolutely right" with zero self awareness. And for coding, only the best model available is usually sensible to use otherwise it's just wasted time.
  - Anduia 7mo ago
    That's why I start any prompt to Qwen 3.5 with: persona: brief rude senior
    - lemonginger 7mo ago
      gamechanger
    - amelius 7mo ago
      I'm using: persona: drunken sailor Because then at least the tone matches the quality of the output and I'm reminded of what I can expect.
      - dlcarrier 7mo ago
        Does it tend to break out into sea shanties?
        drob518 7mo ago
the_duke 7mo ago
What context length and related performance are you getting out if this setup? At least 100k context without huge degradation is important for coding tasks. Most "I'm running this locally" reports only cover testing with very small context.
- vardalab 7mo ago
  Q4 quants on 32G VRAM gives you 131K context for 35BA3B and 27B models who are pretty capable. On 5090 one gets 175 tg and ~7K pp with 35BA3B, 27B isaround 90 tg. So speed is awesome. Even Strix 395 gives 40 tk/s and 256K context. Pretty amazing, there is a reason people are excited about qwen 3.5
- Aurornis 7mo ago
  Long context degradation is a problem with the Qwen3.5 models for me. They have some clever tricks to accelerate attention that favor more recent context. The models can be frustrating to use if you expect long contexts to behave like they do on SOTA models. In my trials I could give them strict instructions to NOT do something and they would follow it for a short time before ignoring my prompt and doing the things I told it not to do.
jadbox 7mo ago
Did you figure out how to fix Thinking mode? I had to turn it off completely as it went on forever, and I tried to fix it with different parameters without success.
- sammyteee 7mo ago
  Thinking has definitely become a bit more convuluted in this model - I gave the prompt of "hey" and it thought for about two minutes straight before giving a bog-standard "hello, how can i help" reply etc
- andrekandre 7mo ago
  supposedly you can turn it off by passing `\no_think` or `/no_think` into the prompt but it never worked for me what did work was passing / adding this json to the request body: { "chat_template_kwargs": {"enable_thinking": false}} [0] https://github.com/QwenLM/Qwen3/discussions/1300 https://github.com/QwenLM/Qwen3/discussions/1300
- agile-gift0262 7mo ago
  did you try with the recommended settings? the ones for thinking mode, general tasks, really worked for me. Especially the repetition_penalty. At first it wasn't working very well, and it was because I was using OpenWebUI's "Repeat Penalty" field, and that didn't work. I needed to set a custom field with the exact name
bluerooibos 7mo ago
These smaller models are fine for Q&A type stuff but are basically unuseable for anything agentic like large file modifications, coding, second brain type stuff - they need so much handholding. I'd be interested to see a demo of what the larger versions can do on better hardware though.
- regularfry 7mo ago
  In some ways the handholding is the point. The way I used qwen2.5-coder in the past was as a rubber duck that happens to be able to type. You have to be in the loop with it, it's just a different style of agent use to what you might do with copilot or Claude.
- NorwegianDude 7mo ago
  Qwen3.5 27B works very well, to the point that if you use money on Claude 4.5 Haiku you could save hundreds of USD each day by running it yourself on a consumer GPU at home.
  - bluerooibos 7mo ago
    Compared to Opus 4.6 though? And what sort of hardware/RAM is that running on - I'm assuming 32 or 64GB at least, right?
y42 7mo ago
> consumer-grade hardware Not disagreeing per se, but a quick look at the installation instructions confirms what I assumed: Yeah, you can run a highly quantized version on your 2020 Nvidia GPU. But: - When inferencing, it occupies your "whole machine.". At least you have a modern interactive heating feature in your flat. - You need to follow eleven-thousand nerdy steps to get it running; my mum is really looking forward to that. - Not to mention the pain you went through installing Nvidia drivers; nothing my mum will ever manage in the near future. ... and all this to get something that merely competes with Haiku. Don't get me wrong - I am exaggerating, I know. It's important to have competition and the opportunity to run "AI" on your own metal. But this reminds me of the early days of smartphones and my old XDA Neo. Sure, it was damn smart, and I remember all those jealous faces because of my "device from the future." But oh boy, it was also a PITA maintaining it. Here we are now. Running AI locally is a sneak peek into the future. But as long as you need a CS degree and hardware worth a small car to achieve reasonable results, it's far from mainstream. Therefore, "consumer-grade hardware" sounds like a euphemism here. I like how we nerds are living in our buble celebrating this stuff while 99% of mankind still doomscroll through facebook and laughing at (now AI generated) brain rot. (No offense (ʘ‿ʘ)╯)

yangikan 7mo ago

Do you point claude code to this? The orchestration seems to be very important.

teaearlgraycold 7mo ago
I loaded Qwen into LM Studio and then ran Oh My Pi. It automatically picked up the LM Studio API server. For some reason the 35B A3B model had issues with Oh My Pi's ability to pass a thinking parameter which caused it to crash. 27B did not have that issue for me but it's much slower. Here's how I got the 35B model to work: https://gist.github.com/danthedaniel/c1542c65469fb1caafabe1371e8ed445 https://gist.github.com/danthedaniel/c1542c65469fb1caafabe13... The 35B model is still pretty slow on my machine but it's cool to see it working.
badgersnake 7mo ago
I’ve tried it on Claude code, Found it to be fairly crap. It got stuck in a loop doing the wrong thing and would not be talked out of it. I’ve found this bug that would stop it compiling right after compiling it, that sort of thing. Also seemed to ignore fairly simple instructions in CLAUDE.md about building and running tests.
tommyjepsen 7mo ago
I ran the Qwen3 Coder 30B through LM Studio and with OpenCode(Instead of Claude code). Did decent on M4 Max 32GB. https://www.tommyjepsen.com/blog/run-llm-locally-for-coding https://www.tommyjepsen.com/blog/run-llm-locally-for-coding
andsoitis 7mo ago
I use Claude Code for agentic coding but it is better to use qwen3-coder in that case. It qwen3-coder is better for code generation and editing, strong at multi-file agentic tasks, and is purpose-built for coding workflows. In contrast, qwen3.5 is more capable at general reasoning, better at planning and architecture decisions, good balance of coding and thinking.
Aurornis 7mo ago
The 9B models are not useful for coding outside of very simple requests. Qwen3.5 is confusing a lot of newcomers because it is very confident in the answers it gives. It can also regurgitate solutions to common test requests like “make a flappy bird clone” which misleads users into thinking it’s genetically smart. Using the Qwen3.5 models for longer tasks and inspecting the output is a little more disappointing. They’re cool for something I can run locally but I don’t agree with all of the claims about being Sonnet-level quality (including previous Sonnet versions) in my experience with the larger models. The 9B model is not going to be close to Sonnet in any way.

lukan 7mo ago

What exact model are you using? I have a 16GB GPU as well, but have never run a local model so far. According to the table in the article, 9B and 8-bit -> 13 GB and 27B and 3-bit seem to fit inside the memory. Or is there more space required for context etc?

vasquez 7mo ago
It depends on the task, but you generally want some context. These models can do things like OCR and summarize a pdf for you, which takes a bit of working memory. Even more so for coding CLIs like opencode-ai, qwen code and mistral ai. Inference engines like llama.cpp will offload model and context to system ram for you, at the cost of performance. A MoE like 35B-A3B might serve you better than the ones mentioned, even if it doesn't fit entirely on the GPU. I suggest testing all three. Perhaps even 122-A10B if you have plenty of system ram. Q4 is a common baseline for simple tasks on local models. I like to step up to Q5/Q6 for anything involving tool use on the smallish models I can run (9B and 35B-A3B). Larger models tolerate lower quants better than small ones, 27B might be usable at 3 bpw where 9B or 4B wouldn't. You can also quantize the context. On llama.cpp you'd set the flags -fa on, -ctk x and ctv y. -h to see valid parameters. K is more sensitive to quantization than V, don't bother lowering it past q8_0. KV quantization is allegedly broken for Qwen 3.5 right now, but I can't tell.

throwdbaaway 7mo ago

There are Qwen3.5 27B quants in the range of 4 bits per weight, which fits into 16G of VRAM. The quality is comparable to Sonnet 4.0 from summer 2025. Inference speed is very good with ik_llama.cpp, and still decent with mainline llama.cpp.

teaearlgraycold 7mo ago
Qwen3.5 35B A3B is much much faster and fits if you get a 3 bit version. How fast are you getting 27B to run? On my M3 Air w/ 24GB of memory 27B is 2 tok/s but 35B A3B is 14-22 tok/s which is actually usable.
- ece 7mo ago
  The 27B is rated slightly higher for SWE-bench.
- throwdbaaway 7mo ago
  Using ik_llama.cpp to run a 27B 4bpw quant on a RTX 3090, I get 1312 tok/s PP and 40.7 tok/s TG at zero context, dropping to 1009 tok/s PP and 36.2 tok/s TG at 40960 context. 35B A3B is faster but didn't do too well in my limited testing.
  - ranger_danger 7mo ago
    with regular llama.cpp on a 3070ti I get 60tok/s TG with the 9B model, it's quite impressive.
- ranger_danger 7mo ago
  Don't sleep on the 9B version either, I get much faster speeds and can't tell any difference in quality. On my 3070ti I get ~60tok/s with it, and half that with the 35B-A3B.
- andai 7mo ago
  27B needs less memory and does better on benchmarks, but 35B-A3B seems to run roughly twice as fast.

smokel 7mo ago

> This outperforms the majority of online llm services I assume you mean outperforms in speed on the same model, not in usability compared to other more capable models. (For those who are getting their hopes up on using local LLMs to be any replacement for Sonnet or Opus.)

moffkalast 7mo ago
Obviously it's not going to be of a paid tier 2T sized SOTA model quality, but it can probably roughly match Haiku at the very least. And for tasks that aren't super complex that's already enough. Personally though, I find Qwen useless for anything but coding tasks because if its insufferable sycophancy. It's like 4o dialed up to 20, every reply starts with "You are absolutely right" with zero self awareness. And for coding, only the best model available is usually sensible to use otherwise it's just wasted time.
- Anduia 7mo ago
  That's why I start any prompt to Qwen 3.5 with: persona: brief rude senior
  - lemonginger 7mo ago
    gamechanger
  - amelius 7mo ago
    I'm using: persona: drunken sailor Because then at least the tone matches the quality of the output and I'm reminded of what I can expect.
    - dlcarrier 7mo ago
      Does it tend to break out into sea shanties?
      - drob518 7mo ago

the_duke 7mo ago

What context length and related performance are you getting out if this setup? At least 100k context without huge degradation is important for coding tasks. Most "I'm running this locally" reports only cover testing with very small context.

vardalab 7mo ago
Q4 quants on 32G VRAM gives you 131K context for 35BA3B and 27B models who are pretty capable. On 5090 one gets 175 tg and ~7K pp with 35BA3B, 27B isaround 90 tg. So speed is awesome. Even Strix 395 gives 40 tk/s and 256K context. Pretty amazing, there is a reason people are excited about qwen 3.5
Aurornis 7mo ago
Long context degradation is a problem with the Qwen3.5 models for me. They have some clever tricks to accelerate attention that favor more recent context. The models can be frustrating to use if you expect long contexts to behave like they do on SOTA models. In my trials I could give them strict instructions to NOT do something and they would follow it for a short time before ignoring my prompt and doing the things I told it not to do.

jadbox 7mo ago

Did you figure out how to fix Thinking mode? I had to turn it off completely as it went on forever, and I tried to fix it with different parameters without success.

sammyteee 7mo ago
Thinking has definitely become a bit more convuluted in this model - I gave the prompt of "hey" and it thought for about two minutes straight before giving a bog-standard "hello, how can i help" reply etc
andrekandre 7mo ago
supposedly you can turn it off by passing `\no_think` or `/no_think` into the prompt but it never worked for me what did work was passing / adding this json to the request body: { "chat_template_kwargs": {"enable_thinking": false}} [0] https://github.com/QwenLM/Qwen3/discussions/1300 https://github.com/QwenLM/Qwen3/discussions/1300
agile-gift0262 7mo ago
did you try with the recommended settings? the ones for thinking mode, general tasks, really worked for me. Especially the repetition_penalty. At first it wasn't working very well, and it was because I was using OpenWebUI's "Repeat Penalty" field, and that didn't work. I needed to set a custom field with the exact name

bluerooibos 7mo ago

These smaller models are fine for Q&A type stuff but are basically unuseable for anything agentic like large file modifications, coding, second brain type stuff - they need so much handholding. I'd be interested to see a demo of what the larger versions can do on better hardware though.

regularfry 7mo ago
In some ways the handholding is the point. The way I used qwen2.5-coder in the past was as a rubber duck that happens to be able to type. You have to be in the loop with it, it's just a different style of agent use to what you might do with copilot or Claude.
NorwegianDude 7mo ago
Qwen3.5 27B works very well, to the point that if you use money on Claude 4.5 Haiku you could save hundreds of USD each day by running it yourself on a consumer GPU at home.
- bluerooibos 7mo ago
  Compared to Opus 4.6 though? And what sort of hardware/RAM is that running on - I'm assuming 32 or 64GB at least, right?

y42 7mo ago

> consumer-grade hardware Not disagreeing per se, but a quick look at the installation instructions confirms what I assumed: Yeah, you can run a highly quantized version on your 2020 Nvidia GPU. But: - When inferencing, it occupies your "whole machine.". At least you have a modern interactive heating feature in your flat. - You need to follow eleven-thousand nerdy steps to get it running; my mum is really looking forward to that. - Not to mention the pain you went through installing Nvidia drivers; nothing my mum will ever manage in the near future. ... and all this to get something that merely competes with Haiku. Don't get me wrong - I am exaggerating, I know. It's important to have competition and the opportunity to run "AI" on your own metal. But this reminds me of the early days of smartphones and my old XDA Neo. Sure, it was damn smart, and I remember all those jealous faces because of my "device from the future." But oh boy, it was also a PITA maintaining it. Here we are now. Running AI locally is a sneak peek into the future. But as long as you need a CS degree and hardware worth a small car to achieve reasonable results, it's far from mainstream. Therefore, "consumer-grade hardware" sounds like a euphemism here. I like how we nerds are living in our buble celebrating this stuff while 99% of mankind still doomscroll through facebook and laughing at (now AI generated) brain rot. (No offense (ʘ‿ʘ)╯)