4 ms·
The gguf is up and works, I don’t know if it’s them or unsloth that’s facilitated this but it’s nice because e.g. Inkling still doesn’t appear to have support i
by andy99 2mo ago
The gguf is up and works, I don’t know if it’s them or unsloth that’s facilitated this but it’s nice because e.g. Inkling still doesn’t appear to have support in llama.cpp which makes it irrelevant to a class of user.
Unfortunately I don’t have enough experience with Qwen 27B to immediately compare, but I do it’s Qwen 3.6 35B A3. It’s much slower obviously but it seems to be way more efficient with its thinking to the point that using it might actually be faster. I find Qwen and some others rehash the same things over and over when thinking without getting anywhere, in mg limited checks here Muse is much better.
- dofm 2mo agoI don't really use the Qwen 3.6 27B though I do test the variants (Bonsai, ThinkingCap). I really like the 3.6 35B A3B for experiments, and it seems OK, but as you say, it spins round in thinking loops more than say the 26B Gemma 4 does. If Muse doesn't actually-wait itself as much it will be very interesting. I am just downloading it to run my small tests.
- jedbrooke 2mo agothe “Actually… But wait!” style responses are so annoying, even Claude opus struggles with this so I’d be interested if meta has done something to cut down on that while still giving good responses
- dofm 2mo agoIt's not perfect but it is very terse! Better than BottleCap managed to do with post-training Qwen in ThinkingCap. I suspect it will help a lot with enabling preserve-reasoning, because the biggest apparent limitation of this model is the 128K context window. Though the practical issue I am seeing on my M1 Max MBP is that performance suddenly drops off a cliff if I have DFlash enabled.
- hadlock 2mo ago128k context window is a complete non-started for us. We need to optimize our most needy agentic jobs, but our average context is well above that
- dofm 2mo agoThe model card does say 131K+ but I have no idea what scope the + really has in practice. Feels like overclocking; you're-on-your-own territory? One thing I will say is that its thinking traces are really quite terse. It genuinely seems to spend many fewer tokens on reasoning. So that might help a bit.
- 0xc133 2mo agoI was able to use rope/yarn scaling with llama.cpp to extend the context window to 256K and it seems to be pretty usable on the debugging-and-bugfixing session I have that’s 216K tokens deep. No infinite output loops, reasoning is still coherent, tool calls appear to be passing and failing at roughly the same rate as a fresh context window. Haven’t tried going to 512K or higher yet but 256K definitely seems usable to me on a single Radeon R9700. I added these arguments to my llama-server call, using the official GGUF release from Meta’s account on HF. `--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 131072 --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144`
- dofm 2mo agoUseful stuff, thank you. And more reading for me to do! (DFlash seems rather weird on my M1 Max; model performance suddenly drops off a cliff like maybe there is some memory management issue).
- hadlock 2mo agoIn my experience I am getting 23-24 t/s output with dflash off, and it craters to ~9 t/s with it on, miss rate exceeding 50%. And I'm using the same device as stated on their model page/card. We might need to wait for the software to catch up
- bitexploder 2mo agoI have a custom A3B proxy that caps its thinking off. It is a known issue with the model that Qwen themselves documented but is almost never addressed in any harnesses. I also patched up a few other known bugs in the proxy. I highly recommend you shim A3B and when it hits 2K thinking tokens inject (paraphrasing) 'Time to wrap it up bud! Get to work' into its thinking stream. It almost always gets to work. If it needs more time to think there is always next turn. In my experience it is almost never productively thinking past that point, just spinning in circles. I also reinject all of the thinking. And there are a few tells that it is getting stuck. I have an optional mode that takes the last few turns and tool calls and shoots it off to DSV4 with a prompt to basically understand where it is at and inject better thinking and or planning. It almost always gets it over relatively difficult humps, but some of the time I don't want things going remote. It might end up with 10-30 cents of DSV4 calls over a hours and the quality improvement is remarkable. The other thing is I trick it into thinking a web_search tool is a web search but it really just asks DSV4 the prompt. DSV4 is a cheap filter to help prevent prompt injection lol. You can give it other models but DSV4 is my cheap-mode default. edit: oh! My final 35B A3B tip -- use an extremely simple harness. Pi is good. Pi's default tools almost exactly match what Qwen says they tested the model with (likely meaning that tool set is also what they trained it with or something similar). So, in my experience bigger harnesses don't have a noticeable improve ment on tasks.
- dofm 2mo agoFascinating, thank you. I am trying to switch to pi from opencode (my own thinking loops and burnout are a challenge lately). It had not occurred to me that you could nudge it to stop thinking with a proxy. Nice idea. Will favourite your comment and come back to it. ETA: Incidentally you've helped me put into words the difference between the way Muse Glimmer thinks to the way Qwen thinks. There is a clear sense of urgency in Glimmer's thinking traces.
- andy99 2mo agoHaving spent a good part of the day with it, glimmer reminds me of Rorschach from The Watchmen. No unessential parts of speech, action oriented, brief and to the point. From a token perspective anyway it’s great, and it seems to hold its own well against more verbose models. I really do feel like it’s effective tok / s is way higher because it doesn’t waste them.
- segmondy 2mo agoThere's an inkling branch, go to unsloth, read - https://unsloth.ai/docs/models/inkling https://unsloth.ai/docs/models/inkling