3 ms·
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much
by johndough 1mo ago
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.
I could of course download and test myself, but that would take days with my internet connection.
- InvertedRhodium 1mo agoI tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer. I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything. After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate. Q4_K_P
- Balinares 1mo agoI anecdotally observed the same. Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.
- danielhanchen 1mo agoWe made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc) We do plan to do larger benchmark suites though!
- johndough 1mo agoGreat to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure. The current benchmark suites that frontier AI labs use are probably a good fit, e.g. https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20comparison%20models https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com... https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20at%20the%20frontier https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20... https://www.anthropic.com/news/claude-opus-5 https://www.anthropic.com/news/claude-opus-5 https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/ But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?
- zdragnar 1mo agoPurely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait... It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
- zenoprax 1mo agoI've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.
- stkdump 1mo agoThere is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.
- lifepillar 1mo agoAccording to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more). [0] https://m.youtube.com/watch?v=z64J6bC16iQ https://m.youtube.com/watch?v=z64J6bC16iQ
- zenoprax 1mo agoGreat, thanks! I'll try it out.