4 ms·
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it rea
by lnenad 1mo ago
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
- grim_io 1mo agoThat's low reasoning for a model, but max for a HN comment.
- cyanydeez 1mo agoMy stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat. Also, heating my home during the winter is nice. Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
- lnenad 1mo agoYeah 122B is the sweet spot for me as well. Even deepseek flash overthinks on stuff way too much. I think they fully rely on large reasoning turns to achieve better quality. The result of course means we wait a long time to get results even with high throughput as a lot of tokens are wasted.
- redrix 1mo agoYou’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: 1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment. One final thing worth mentioning: (I made myself irrationally angry writing this)
- gorgmah 1mo agoYou made me irrationally laugh reading this
- lsb 1mo agoThis is glitch art for text, I love it
- raducu 1mo agoOn one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?
- samsari 1mo ago70% of the posts on HN are already satire and performance art
- pixelpoet 1mo agoAnd full of made up statistics.
- NamlchakKhandro 1mo agoWhich then devolve into supporting arguments for communism. Themselves becoming food for future irrational anecdotes about communism.
- switchbak 1mo agoMore than 2 levels and out come my downvotes. Or if it's just knee jerk with zero humour. But I probably violate my own rules ... which is to be expected.
- terrut 1mo agoYou are right to push back— Sorry, couldn't resist ;) I agree that this is not what we normally come here for, but this thread made me chuckle. I think we are just venting our shared frustrations a bit.
- esafak 1mo agoIt will be interesting to see the token efficiency analysis. This is my first question now with Chinese models; I take raw benchmark performance for granted.
- dannyw 1mo agoYou might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more. For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 https://arxiv.org/abs/2607.22925 for some starters. Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
- lnenad 1mo agoYeah I understand, it's my assumption that the actually/wait/but have a point. It doesn't reduce the fact that it increases the time for tasks substantially.
- mdp2021 1mo ago> For example, even if you make thinking tokens literally just Generally speaking yes, but actually no (just randomness is suboptimal, adding steps just to add steps is suboptimal). There is a mechanism working there (in having a CoT) that is not quite clear. The task is to optimize the efficiency of CoT. Understanding that it is not a plain "chain of thought" is the start of the problem, the solution is not there yet. If we had the solution, there would exist no overthinking - CoT would be optimal (lean and essential plus best results).
- xatnys 1mo agoDid you observe the model overthinking on practical tasks? While 3.8 does think a lot on xhigh I've found that it really depends on the task. On one-shot prompts that are usually the first to be posted during new releases it will tend to spend a lot more time thinking than doing. In other words the more open ended a problem space becomes, the more Qwen will tend to second-guess itself. Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
- lnenad 1mo agoEspecially on practical tasks. One shot prompts work better at Q6_K_XL for me. It loads a file, then analyses then second guesses itself then again then again then it tries to come up with a solution then second guess rinse and repeat. 122b is the perfect balance but it lacks quality for harder to solve stuff. I've ran DS Flash 0731 at Q4KXL, 3.8 Q6KXL, GLM 5.2 Q4KXL and they all over-reason. At least that's how it looks like to me when comparing with frontier models, even weaker ones.
- overgard 1mo agoYeah, I ran into an overthinking loop with it a couple days ago on a task that shouldn't have been that hard. (It's kind of interesting to watch the internal conversation happening with it). Overall I'm impressed with it, but setting the /effort to medium is what you usually want (it defaults to xhigh). I do wonder if I had made it write out a plan if I would have avoided that though.
- dofm 1mo agoYes. xhigh can not just overdo the answer, it can also trip itself up and end up writing worse code. Even in the lower reasoning levels I find I want to like Qwen 3.8 27B and mostly don’t; it’s OK in the low reasoning effort, though. Muse Glimmer is the one I actually enjoy working with, at least so far. But I am trying to use it more as a sidekick than as a long horizon developer, because that is a better fit for how I want to use AI, and it appears to have been well trained for that.
- atmosx 1mo agoWhat about adding rtk proxy?
- Casteil 1mo agoYep.. for 'general purpose' use I found qwen3.8:27b to be disappointing due to overthinking. It's brutal especially considering how slow it is compared to MoE variants. It often overthinks to the magnitude of ~10x the tokens vs a ~4x faster gemma4:26b-a3b. As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer. Obviously, 'YMMV' depending on your use case... just sharing my two cents.
- SillyUsername 1mo agoI use medium generally, that's about a minute at 20t/s and off for general chat (few seconds for a response). What kind of setup are you running it on?
- marcthor 1mo ago[flagged]
- javier2 1mo agoYou are absolutely right to push back on this. Let me think for a moment.
- giancarlostoro 1mo agoSince you're running through the trouble of setting that up, if its 125B params, but only 6B is activated, does that mean you mainly need to allocate enough VRAM for that much of the model? Or do you still need enough VRAM for the whole thing (and buffer for context window)? Or maybe anyone can inform me, this is one area I'm uninformed in.
- hgoel 1mo agoI believe that at minimum, for usable performance, you need to be able to hold the 125B params + 51B ngrams in some sort of RAM. Ideally VRAM, but the benefit of the MoE design is better performance with unified memory since most of that RAM is not read for every single token. So you could potentially have the model loaded in CPU RAM, and let unified memory systems page the relevant chunks on demand to VRAM, or run on a fully unified memory system and be able to achieve good speeds even with the limited memory bandwidth most of them have.
- petu 1mo agoYou need VRAM for the whole thing for optimal performance. Activation is chosen "randomly" for each token. PCIe becomes bottleneck, so much that just doing computation on CPU is likely faster. But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).
- looksjjhg 1mo agoWhat kind of machine do you have in your homelab that can run this model?!
- petu 1mo agoThis is needs ~80GB of fast memory at 4 bits per weight. Faster memory is better, but probably even something like 3090 + 64GB RAM should work (not fast, but maybe even 20-30 t/s? llama.cpp support pending).
- lnenad 1mo agoI've got a 48c Epyc with 2x3090s and 512gb ddr4 3200. It's good enough for 25+ tps with deepseek so I'm hoping for similar performance with less overthinking.