4 ms·
Chinese labs are so used to manipulating benchmarks to try to flatter inferior models that when they finally have one that's really pretty good I think the offi
by mrngld 1mo ago
Chinese labs are so used to manipulating benchmarks to try to flatter inferior models that when they finally have one that's really pretty good I think the official announcement here undersells it.
https://deepswe.datacurve.ai/ https://deepswe.datacurve.ai/
That's pretty solid. Smarter and cheaper than Luna xhigh, not as smart but less expensive than Luna max. Smashes deepseek v4 flash, and even worse it matches v4 pro at a tiny fraction the cost. Roughly equivalent to sol medium, at a fraction the cost.
They should've just lead with real, up to date data, because it's good, not the silly old tactics like comparing to Opus 4.8 when 5.0 is out in many of their charts.
Congrats to them!
- stavros 1mo agoOpus 5 is better than Fable in this benchmark?
- zarzavat 1mo agoEven Artificial Analysis has Opus 5 better than Fable in their aggregated "Intelligence Index" which combines 9 benchmarks. Opus 5 is heavily benchmaxxed.
- nijave 1mo agoDepends on the benchmark but yes. I think Opus is more heavily optimized for coding. On the usability side, its output is almost intolerable to read. It seems to code fairly well. Fable is more enjoyable to use for planning/interacting with
- seaal 1mo agoOnly 73K output tokens too. Anthropic should really be embarrassed with their Sonnet 5 price/performance.
- redox99 1mo agoIt's also better than Sol (at whatever effort) at designing pretty UIs. I have a Codex sub and I've been using this model for UI stuff.
- mkagenius 1mo ago> I've been using this model for UI stuff. The flash one?
- redox99 1mo agoYeah, when it was secretly called Ox Alpha.
- glub 1mo agoYes. I have no UI experience, and wanted a model that could produce something good without me telling it how anything should look like. My prompt was something like: "here's data I have, here's what matters to me, create HTML mockup". All GPT 5.6 models were laughably bad. And I don't want to downplay it - they were just absolutely, objectively horrible. Every single attempt was what I could probably call "if json was ui". Claude models produced... "claude look". GLM 5.3 - somewhere between GPT and Claude. Kimi k3 - each attempt produced beautiful UIs. It used components that I didn't even know existed and wouldn't even know to ask for. But expensive, very expensive. ox-alpha (GLM 5.3 flash) was very close to K3. And at this price point, it's already configured as "designer" model in my oh-my-pi.
- redox99 1mo agoYeah, exact same for me. K3 used to be my go to for UI but it's quite expensive. Ox alpha being so cheap and so comparatively good at design is crazy. I haven't tried it but I think Qwen Max is also very good at design.
- re-thc 1mo ago> They should've just lead with real, up to date data, because it's good, not the silly old tactics like comparing to Opus 4.8 when 5.0 is out in many of their charts It's what people know. Opus is just the common target. > Smarter and cheaper than Luna xhigh, not as smart but less expensive than Luna max. Smashes deepseek v4 flash The problem with this and DeepSWE is it goes for a very specific profile. I'm not convinced DeepSWE is any accurate in actual work. It's surely a different signal (compared to some that allow cheating) but it has its own issues, e.g. weak harness. Luna is great at following instructions but bad instructions or anything not covered = death. Deepseek is more analytical. Good for bug tracking. GLM is a better all rounder in some ways. Better at creativity.
- s777 1mo ago> I'm not convinced DeepSWE is any accurate in actual work. They listed Muse Spark 1.2 around DeepSeek V4 Flash even though it's a much shittier model in basically every aspect. > GLM is a better all rounder in some ways. Better at creativity. I agree with the creativity part.
- cameronh90 1mo agoMaybe others have found otherwise, but I find the benchmarks drastically different to real world "feel" of a model, even within the same harness. I'm not sure if this just reflects personal interaction styles, or if it is indicative of benchmaxxing or unrealistic automated benchmarking methodology. Opus 5 consistently comes at or near the top, but outputs constant unreadable jibberish. Meanwhile GPT 5.6 Luna medium tends to be rated pretty poor on agentic tasks compared to the Chinese lab open models, but I find the latter much more likely to lose track of their own behaviour during a long-horizon task or get stuck in a doom loop. (This isn't a comment on GLM-5.3 Flash as I've not used it!)
- trey-jones 1mo agoI've been using 5.3 since they initially announced it and my gut feel is that it's not as good as 5.2 for agentic tasks. I'm still using it - I don't think it's bad. I'm just not convinced it's better.
- disiplus 1mo agoidk i think that i spend significant tokens with both to be able to tell 5.3 is way better overall. https://kommodo.ai/i/IFSUUQYT522uZXWvePZ1 https://kommodo.ai/i/IFSUUQYT522uZXWvePZ1 it still fells stupid sometimes and it is benchmaxxed for sure. but its good enough that im building all the hobby projects with it.
- gunalx 1mo agoI have had the opposite where I felt 5.3 as a stronger model than 5.2. Its feels way more in tune with my code, and does more nuanced edits. Though I do handhold my models a lot, so might fall outside the agentic term.
- tyre 1mo agoOpus is a yap god. I've found it much, much better with Claude Code's output style set to `Concise` and this: https://news.ycombinator.com/item?id=49413456 https://news.ycombinator.com/item?id=49413456 We shouldn't have to resort to this, but it can be mitigated enough that it stays as my daily worker agent. Although, I mostly use Fable to farm out to Opus agents so I don't have as much exposure to what kind of blathering is going on in there.
- glub 1mo agoI don't know how anyone can actually use Luna max on ANY real workload. I've had Sol orchestrate a bunch of Luna agents, these agents were explicitly given small chunks of larger objectives and they still filled their entire context windows with just reasoning tokens, until compaction hit, and then reasoning again. I've probably wasted a good 40% of my weekly usage on Luna Max agents just thinking and not writing a single line of code.
- amluto 1mo agoThe one time I tried asking Sol to use subagents for a small project, it took a surprisingly long time, used up the entire usage limit in one go, and basically failed the project. I’m pretty sure that plain Sol, serially, could have finished the task faster, cheaper, and far more accurately. I’m also pretty sure that any competent subagent orchestration could have gotten it done with even very simple subagents quickly and cheaply. (Is it really that hard to set up a handful of subagents that all use the same initial context and to load that context with what actually matters? The APIs certainly support it.)
- dannyw 1mo agoOpenAI has been doing wonky stuff with subagents, including encrypting the prompts sent to subagents in Codex. Who knows what’s really going on.
- GreenWatermelon 1mo agoI'm in the same boat. I haven't found sub agents flows useful.
- John7878781 1mo agoLuna max is all I use. In my experience, it works really well for overnight tasks.
- kilroy123 1mo agoI feel the same. I just use it for planning and chatting. Not real coding work.
- 1mo ago