3 ms·

Geometric mean of all these benchmarks : * GPT-5.6 Sol: 65.5 * Fable 5 (w/ fallback): 64.5 * Opus 5: 64.0 * DS-V4-Pro 0813: 62.5 * Kimi-K3: 62.3 * DS-V4-F

Geometric mean of all these benchmarks : * GPT-5.6 Sol: 65.5 * Fable 5 (w/ fallback): 64.5 * Opus 5: 64.0 * DS-V4-Pro 0813: 62.5 * Kimi-K3: 62.3 * DS-V4-Flash 0731: 55.8 * GLM-5.2: 47.3

svachalek 2mo ago
Maybe it's me but I don't see how DS Flash is better than GLM at all, much less by a huge gap. I'd probably protest less against Fable and Opus being put at the same level than many would, but there's no denying the two models are a very different experience from each other. I guess where I'm going is no one should pick a model by the benchmarks.
- platinumrad 2mo ago
  I think instruction following carries outsized weight in these evaluations.
- spijdar 2mo ago
  I'm not the most LLM-savvy person around, and I'm not gonna say I've put a ton of effort into practically compared these open models. But, a month or two ago I did do some "practical evaluates" testing GLM 5.2 versus DSv4 (flash/pro) with OpenCode's subscription with some late 80s Unix clone-type work, and this jives with my experience. GLM ended up being far slower, and far more expensive, for approximately the same results. There was never a problem that GLM could solve that DS couldn't solve, faster, and significantly cheaper. I strongly agree that you shouldn't pick a model based on benchmarks. But for me, I found GLM really underwhelming given its cost and speed. DSv4 isn't as good as GPT or Claude or what have you, but it's fast, and pretty darned effective. I can run a 3-bit quant of DSv4 locally on my system with ~15 tokens per second, and for a local model it might be the most overall effective at coding. For what it is, it's extremely impressive.
  - ApolloFortyNine 2mo ago
    My experience is the same. Imo it has a lot to do with you/the harness tries to get it to test itself. Deepseek v4 flash seems more than capable of understanding when something has failed, and making changes until it works. I've definitely seen it make mistakes I would expect something like Opus to find, but it works through them on it's own (and for literal pennies). At the end of the day, I think that's one of the most important features of a model.
  - wut42 2mo ago
    Exactly the same experience. I really loved GLM5.2 for a while, but after trying it again after riding DSv4 flash (new) for a while, they're mostly at the same capabilities, with GLM being slower and much, much more expensive. A task cost me 2$ where it did very wrong, whereas Flash nailed it almost instantly for like a rounding error on my billing page.
  - kaeluka 2mo ago
    Interesting! GLM 5.2 is slower for sure (although they offer a fast version), and it's more expensive. But in my experience, it's universally better than Deepseek V4-flash-0731. Don't get me wrong, the new Flash version is amazing. But the use cases I have looked at are about source code understanding, bug finding, etc. - GLM 5.2 is clearly better. I think by using some prompt engineering, you will probably be able to close this gap, but some extra work is needed. And I'll say it again: the new Flash version is amazing. I love it. That level of intelligence for the price is unprecedented, and the fact that it's open weights and runs locally makes me genuinely happy.
- spiffytech 2mo ago
  In my little social circle DS4F generally substitutes for GLM 5.2 except it's the next best thing to free.
- segmondy 2mo ago
  It isn't. I run both at home. GLM5.2 Q4 crushes DSv4Flash0731 Q8. I reach for DS for speed and for medium effort level work. If I care about quality I'll reach for GLM5.2 Looking at this release, I'm comparing it to GLM5.2 and it seems to beat GLM5.2, only time/experience will show. If true, then I'm happy. It's much easier to run than Qwen3.8/KimiK3

Geometric mean of all these benchmarks : * GPT-5.6 Sol: 65.5 * Fable 5 (w/ fallback): 64.5 * Opus 5: 64.0 * DS-V4-Pro 0813: 62.5 * Kimi-K3: 62.3 * DS-V4-Flash 0731: 55.8 * GLM-5.2: 47.3

svachalek 2mo ago

Maybe it's me but I don't see how DS Flash is better than GLM at all, much less by a huge gap. I'd probably protest less against Fable and Opus being put at the same level than many would, but there's no denying the two models are a very different experience from each other. I guess where I'm going is no one should pick a model by the benchmarks.

platinumrad 2mo ago
I think instruction following carries outsized weight in these evaluations.
spijdar 2mo ago
I'm not the most LLM-savvy person around, and I'm not gonna say I've put a ton of effort into practically compared these open models. But, a month or two ago I did do some "practical evaluates" testing GLM 5.2 versus DSv4 (flash/pro) with OpenCode's subscription with some late 80s Unix clone-type work, and this jives with my experience. GLM ended up being far slower, and far more expensive, for approximately the same results. There was never a problem that GLM could solve that DS couldn't solve, faster, and significantly cheaper. I strongly agree that you shouldn't pick a model based on benchmarks. But for me, I found GLM really underwhelming given its cost and speed. DSv4 isn't as good as GPT or Claude or what have you, but it's fast, and pretty darned effective. I can run a 3-bit quant of DSv4 locally on my system with ~15 tokens per second, and for a local model it might be the most overall effective at coding. For what it is, it's extremely impressive.
- ApolloFortyNine 2mo ago
  My experience is the same. Imo it has a lot to do with you/the harness tries to get it to test itself. Deepseek v4 flash seems more than capable of understanding when something has failed, and making changes until it works. I've definitely seen it make mistakes I would expect something like Opus to find, but it works through them on it's own (and for literal pennies). At the end of the day, I think that's one of the most important features of a model.
- wut42 2mo ago
  Exactly the same experience. I really loved GLM5.2 for a while, but after trying it again after riding DSv4 flash (new) for a while, they're mostly at the same capabilities, with GLM being slower and much, much more expensive. A task cost me 2$ where it did very wrong, whereas Flash nailed it almost instantly for like a rounding error on my billing page.
- kaeluka 2mo ago
  Interesting! GLM 5.2 is slower for sure (although they offer a fast version), and it's more expensive. But in my experience, it's universally better than Deepseek V4-flash-0731. Don't get me wrong, the new Flash version is amazing. But the use cases I have looked at are about source code understanding, bug finding, etc. - GLM 5.2 is clearly better. I think by using some prompt engineering, you will probably be able to close this gap, but some extra work is needed. And I'll say it again: the new Flash version is amazing. I love it. That level of intelligence for the price is unprecedented, and the fact that it's open weights and runs locally makes me genuinely happy.
spiffytech 2mo ago
In my little social circle DS4F generally substitutes for GLM 5.2 except it's the next best thing to free.
segmondy 2mo ago
It isn't. I run both at home. GLM5.2 Q4 crushes DSv4Flash0731 Q8. I reach for DS for speed and for medium effort level work. If I care about quality I'll reach for GLM5.2 Looking at this release, I'm comparing it to GLM5.2 and it seems to beat GLM5.2, only time/experience will show. If true, then I'm happy. It's much easier to run than Qwen3.8/KimiK3