3 ms·

I wonder if this can be fixed with LORAs.

by javchz 2mo ago

bitexploder 2mo ago
I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.
- dofm 2mo ago
  Unfortunately in xhigh reasoning effort it will burn through 2K tokens before it has even finished its bullet point overview. It really is intense and obsessive. You might need ten times more! Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping). Not seen looping in the “low” reasoning effort mode.
  - bitexploder 2mo ago
    I have been using Muse Glimmer for a few days instead of A3B. It gets the job done quicker than A3B despite being several times slower.
    - dofm 2mo ago
      Yes — I just found out that you can set reasoning level in the prompt — like with Qwen 3.8 27B it is actually really pretty solid at "Reasoning level: low". Ten to thirteen tokens per second on my M1 Max (might be some room to improve this) but it indeed solved as fast as the Qwen 35B. 40 seconds faster on one of my tests that involves three steps. This is very striking.
- logicallee 2mo ago
  >I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. that is amazing, thanks for sharing.
- permalac 2mo ago
  Could you share the proxy and config ? This sounds good.
CapsAdmin 2mo ago
This morning I tried experimenting with this ThinkingCap lora I found someone made for 3.6 https://huggingface.co/signsur4739379373/Qwen3.6-27B-ThinkingCap-LoRA https://huggingface.co/signsur4739379373/Qwen3.6-27B-Thinkin... ThinkingCap is a 3.6 27b finetune that claims to halve thinking tokens while maintaining the same output quality. I've used the model a lot and I'd say it holds up. Since 3.6 has the same architecture as 3.8, the lora can be applied. With the prompt "create a fancy circle in html", these are the results for xhigh, medium, low and xhigh + thinkingcap lora https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78117f79 https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78... (Note that the gists are bloated because they contain the full chat and launch params in text/plain script tags for transparency) I'd say xhigh looks a little better than xhigh + lora, but the lora variant has 40% less thinking tokens. Both seemed to take the same approach with adding random details that weren't explicitly specified. Medium and low (no lora) are close to each other but are much simpler results. This is just me testing a single turn. I haven't tested this on multi turns and whatnot, but I thought the result was interesting enough to share anyway.
- CapsAdmin 2mo ago
  "Generate an SVG of a pelican riding a bicycle" tests: https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 33170 tokens https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh_thinkingcap_30.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 18125 tokens https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh_thinkingcap_35.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 12960 tokens Scale 35 felt a bit noisy and incoherent, but 30 seemed nice. (they use the same seed, but idk how reliable seed in llamacpp is) I use a python test script that captures the answer and renders it to a html page along with the llama-cli log, launch parameters, the chat log, and the python script itself for maximum transparency. :)

I wonder if this can be fixed with LORAs.

by javchz 2mo ago

I wonder if this can be fixed with LORAs.

I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.

dofm 2mo ago
Unfortunately in xhigh reasoning effort it will burn through 2K tokens before it has even finished its bullet point overview. It really is intense and obsessive. You might need ten times more! Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping). Not seen looping in the “low” reasoning effort mode.
- bitexploder 2mo ago
  I have been using Muse Glimmer for a few days instead of A3B. It gets the job done quicker than A3B despite being several times slower.
  - dofm 2mo ago
    Yes — I just found out that you can set reasoning level in the prompt — like with Qwen 3.8 27B it is actually really pretty solid at "Reasoning level: low". Ten to thirteen tokens per second on my M1 Max (might be some room to improve this) but it indeed solved as fast as the Qwen 35B. 40 seconds faster on one of my tests that involves three steps. This is very striking.
logicallee 2mo ago
>I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. that is amazing, thanks for sharing.
permalac 2mo ago
Could you share the proxy and config ? This sounds good.

CapsAdmin 2mo ago

This morning I tried experimenting with this ThinkingCap lora I found someone made for 3.6 https://huggingface.co/signsur4739379373/Qwen3.6-27B-ThinkingCap-LoRA https://huggingface.co/signsur4739379373/Qwen3.6-27B-Thinkin... ThinkingCap is a 3.6 27b finetune that claims to halve thinking tokens while maintaining the same output quality. I've used the model a lot and I'd say it holds up. Since 3.6 has the same architecture as 3.8, the lora can be applied. With the prompt "create a fancy circle in html", these are the results for xhigh, medium, low and xhigh + thinkingcap lora https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78117f79 https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78... (Note that the gists are bloated because they contain the full chat and launch params in text/plain script tags for transparency) I'd say xhigh looks a little better than xhigh + lora, but the lora variant has 40% less thinking tokens. Both seemed to take the same approach with adding random details that weren't explicitly specified. Medium and low (no lora) are close to each other but are much simpler results. This is just me testing a single turn. I haven't tested this on multi turns and whatnot, but I thought the result was interesting enough to share anyway.

CapsAdmin 2mo ago
"Generate an SVG of a pelican riding a bicycle" tests: https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 33170 tokens https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh_thinkingcap_30.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 18125 tokens https://gistpreview.github.io/?815466e3208746488d47679949b68211/xhigh_thinkingcap_35.html https://gistpreview.github.io/?815466e3208746488d47679949b68... - 12960 tokens Scale 35 felt a bit noisy and incoherent, but 30 seemed nice. (they use the same seed, but idk how reliable seed in llamacpp is) I use a python test script that captures the answer and renders it to a html page along with the llama-cli log, launch parameters, the chat log, and the python script itself for maximum transparency. :)