7 ms·
The cost per task chart is telling me that I should _never_ use Sonnet 5 above medium effort level - Opus always performs better for a given cost. So I guess th
by doctoboggan 4mo ago
The cost per task chart is telling me that I should _never_ use Sonnet 5 above medium effort level - Opus always performs better for a given cost. So I guess the takeaway is that if Sonnet 5 medium isn't good enough for you, switch models, not effort levels.
- Torkel 4mo agoYeah, I was looking at the same chart and was very surprised at where the curve is relative to opus... Feels like sonnet 5 is "what if opus had an extra-low effort level"?
- johnfn 4mo agoThat's just one benchmark, though. Tab to the next one and Sonnet 5 performs better as effort goes up just as you'd expect. I imagine the suggestion is that performance vs effort tradeoff is task dependent.
- energy123 4mo agoNo it doesn't? It's worse than Opus across the whole shared frontier on both plots.
- acchow 4mo agoAgreed. The graphs clearly show that opus 4.8 performs strictly better at the same cost per task
- jsnell 4mo agoBut they don't show "strictly better" performance at cost per task! The graphs show parts of the cost/performance pareto frontier occupied by Opus 4.8 and others occupied by Sonnet 5.0. If Opus 4.8 was strictly better at cost per task like you say, by definition the entire frontier would be occupied by Opus. So neither is pareto-dominant over the other. In contrast, Sonnet 5.0 is Pareto-dominent over Sonnet 4.6 on those graphs.
- energy123 4mo ago> by definition the entire frontier would be occupied by Opus. But the entire frontier is occupied by Opus under any reasonable interpolation scheme (piecewise linear which is what they've done, and most reasonable spline or polynomial fits would also lead to the same result) over the overlapping x values for which both are defined. Under that interpolation scheme, for x > ($ cost of Opus low effort), Opus is Pareto-dominant over Sonnet 5. You can see this by picking any point on Opus's interpolation and realizing that you get strictly worse by switching to Sonnet for the same x value or the same y value. Meaning if you want to pay the same $x then you get a worse y, or if you want the same y you pay more $x.
- jsnell 4mo agoI really don't get what you're proposing. The cost ranges do not overlap at the low end. You can't (by definition!) interpolate outside of the range. If you mean extrapolate, at that point you're just making up data. The available effort levels are discrete and covered totally by the benchmarks. You can draw on the monitor with a sharpie to show a "ultra-low" effort level for Opus that scores better than Sonnet "low" at the same price, but it doesn't magic the ultra-low effort into actual existence. (Anyway, the blog post now has an errata and a graph that shows substantially better relative performance for Sonnet 5.0 than the original graph.)
- energy123 4mo agoThat's why I said "over the shared frontier" in my first post and more precisely in my second post I said "over the overlapping x values for which both are defined." It was a claim that applies to a range of x-values where both curves are defined. Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region. Which is what I understand to be your point?
- jsnell 4mo agoThe post I was replying to said "performs strictly better at the same cost per task". That claim was obviously not true, there are costs where Opus cannot do the task and Sonnet can, so Opus can't be performing strictly better that the same cost. It seems that you agree that it is not true. You could make it true by artificially dropping some of the data points, but, like, why? (Again, this is moot given the updated graph.) > Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region. Not so! It's only sound to do that at the low end of the cost axis (x) or the high end of the performance axis (y). You can't do it at the low end of the performance axis or the high end of the cost axis.
- energy123 4mo agoThe arguable caveat is Sonnet may run faster (although this isn't known for sure, due to more tokens being used for the same task), so you can potentially get more done in a synchronous iterative workflow I don't really believe this however, because so much time is spent fixing up after models, that a slower but more intelligent model is a net time saver in my experience.
- kolinko 4mo agoFrom my benchmarks, sadly, it doesn't seem to be the case much. Surprisingly. I found Sonnet comparable in speed to Opus (sic), but perhaps I was testing it wrong?
- riverbirch 4mo agoI can confirm this, I too I'm not seeing much of a difference in practice
- usef- 4mo agoGiven the chip shortage, I wonder if they push Sonnet to the older chips. It presumably requires less memory.
- ZeWaka 4mo agoIt's very interesting. Why even release a new product that underperforms at the same price level? Why not just lock it? I guess it's probably a lot cheaper for them to run, and it cuts costs for them. Seems disingenuous, though.
- intellijdd 4mo agoI noticed that as well but with the introductory pricing, I wonder how true that is. It would be great to see these charts with the promotional pricing just because it’s here for about two whole months. I guess I could get Sonnet 5 to do it.
- AquinasCoder 4mo agoWhile I appreciate, they publish this information, it's increasingly hard to keep track of it all. I've lost the mental model of how different models at different effort levels perform and what tasks they are good at. In practice, I tend to just use the default on Claude Code that works well enough. But I wonder to what degree other users really play around with these settings to optimize for their project.
- jacooper 4mo agoJust use deepswe as a reference point.
- sanderjd 4mo agoWhat I want is a harness that knows how to optimize this kind of thing for me.
- manojlds 4mo agoWhich is your own harness and your own evals for your tasks I guess
- sanderjd 4mo agoMaybe. But that sounds like a large amount of bespoke work for what seems like a common problem?
- manojlds 4mo agoI was talking about enterprise agents and then realized the question is more about coding agents.
- sanderjd 4mo agoAh I see! Yes, I was talking about a coding harness, not an enterprise agent. I entirely agree with you that your suggestion of driving it via evals is the right thing for that use case!
- 2001zhaozhao 4mo agoThere are two wrinkles to this: - For Claude.ai subscriptions I think Sonnet is much cheaper than Opus. This is why there was a "Sonnet only" usage bar for Max tier for the longest time. - For some tasks the sheer amount of raw input tokens is the most important. For example multimodal computer use tasks. You can't make them any more efficient on Opus by turning down the reasoning, so a cheaper model like Sonnet is useful for them
- timcobb 4mo ago> This is why there was a "Sonnet only" usage bar for Max tier for the longest time. it's still there. I still don't totally grok why I can't use all my tokens on Sonnet if I want to... maybe that signals something?
- laughingcurve 4mo agoDistillation attacks? Volume of calls?
- i000 4mo agoThey want to encourage diversifying model use.
- manojlds 4mo agoOpus 4.8 high doing better and cheaper than Sonnet 5 xhigh
- seiru 4mo agoWorth noting that the default chart there is for "agentic search performance", not coding. I didn't see an effort comparison for coding specifically.
- al_borland 4mo agoWhat is a "task" in real-world terms? If it will be $15/million output tokens, and high/xhigh is somewhere in the $7.50/task range. Does that mean a single task is using 500k tokens. That seems like it would start to add up fast.
- wyre 4mo agoI’ve found input tokens is around 5x more than output, so a task could be a couple million thinking tokens and then a few couple 100k output tokens?
- booi 4mo agoi actually exclusively use Sonnet in low effort level. It's too slow otherwise and at a higher effort levels is strictly worse than Opus.
- lucamark 4mo agoYou're referring to the Agentic search, but if you look at the Agentic computer use the cost is basically halved. However, I am also confused about market positioning. Too expensive to perform daily tasks - open souce models are much cheaper - and not frontier model to address complex real world problems. Rarely used Sonnet btw.
- energy123 4mo agoYou're the second person that has said this but I cannot understand why you are interpreting the "Agentic computer use" graph in this manner. The graph shows that Opus is cheaper than Sonnet for the same performance. Unless I am suffering a cognitive blindness thing right now.
- lucamark 4mo agoWrong! Look at it better. It shows that Opus has superior performance but at higher cost.
- doctoboggan 4mo agoNo, you are misunderstanding the graph. Draw a vertical line anywhere, that is a "constant cost" line. For any given cost, Opus 4.8 has a higher performance than Sonnet 5. Only where Sonnet 5 effort is at medium or low would it make any sense to use it, as there isn't even an equivalent Opus effort level to compare to. Alternatively you can draw a horizontal "constant performance" line and see that Opus is cheaper for a given performance level.
- energy123 4mo agoNo, that's apples and oranges. You need to compare Sonnet5's 79% with the interpolated Opus4.8's 79%.
- brokencode 4mo agoThat is a bad comparison. Compare Sonnet xhigh against Opus medium, which is both better and cheaper.
- nicce 4mo ago> Opus always performs better for a given cost. Assume it to get deprecated sooner rather than later.
- Natelinathan 4mo agoI just re-wrote the /code-review skill anthropic ships to use Sonnet 4.6 for some tasks as it was using Opus for simple git diff commands and similarily mechanical tasks (launched 100+ agents for one of my diffs, cmon). I wonder how Sonnet 5 will impact my usage. Does anyone else have any review token saving measures?
- goldenarm 4mo agoIt's funny the exact same thing happened to Gemini 3.5 flash. Cheaper and more agentic model that ends up worse and more expensive than 3.5 pro low.
- XCSme 4mo agoWell, it is a Sonnet model, it is indeed better[0] than Sonnet 4.6 (smarter, faster, cheaper), but I don't see why would you use it as opposed to Opus 4.8 low or GLM-5.2... [0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-medium/anthropic-claude-sonnet-5-medium/ https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-med...
- XCSme 4mo agoWhat's interesting, is that Sonnet 5 is actually worse[0] than 4.6 without reasoning. It makes some sense, as models are trained more and more with reasoning, than without. [0]: https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-none/anthropic-claude-sonnet-5-none/ https://aibenchy.com/compare/anthropic-claude-sonnet-4-6-non...
- partsch 4mo agoI feel like the charts have been adjusted. I am quite sure, they looked different a couple hours ago...
- callahad 4mo agoThey've absolutely both changed. The initial version I saw didn't include max effort data points on the first chart, and the plot itself was much less favorable to Sonnet at high/xhigh relative to Opus, but the new chart shows them as closer competitors. Weird.
- make3 4mo agoit might be worth it if speed is an issue
- windexh8er 4mo agoExcept for the fact that Opus 4.8 is not good. Constant hallucinations, doesn't use the web very intentionally until you explicitly ask it to and it nopes out rather quick on benign items. Anthropic has been very disappointing as of late. All of the gatekeeping is taking a toll on what should be some of the better models out there, but you can't trust 4.8 to go off on its own. It will burn down tokens doing what it deems correct as per its guidance. Truly painful to use.
- lukan 4mo ago"but you can't trust 4.8 to go off on its own." And what (avaiable) model do you trust to go off on its own?
- windexh8er 4mo agoThe point is Anthropic has advertised their models in this way. There are plenty of models that can be used in long running situations that have proven to be more capable. Opus 4.8 is not that, and ironic given it's their top public model.
- jimbo808 4mo agoThey're actively trying to use lobbying power to make open weight models illegal. So I'm just not going to use their services at all anymore. I don't think they're a net gain if you're a skilled senior, and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug. I'll be okay without their bullshit generator.
- andsoitis 4mo ago> They're actively trying to use lobbying power to make open weight models illegal. What is your evidence?
- jimbo808 4mo agohttps://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_testimony_-_amodei.pdf https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_... https://xcancel.com/coinbureau/status/2071330294452666695 https://xcancel.com/coinbureau/status/2071330294452666695 https://www.techpolicy.press/transcript-senate-hearing-on-principles-for-ai-regulation/ https://www.techpolicy.press/transcript-senate-hearing-on-pr... > "Once the weights of a model are public, they cannot be retrieved. If a model possesses dangerous capabilities, it is permanently out in the wild... We need to consider regulatory frameworks that account for the unique risks of open-source distribution of highly capable frontier models."
- TurdF3rguson 4mo agoThat's true I guess. If someone decides a model needs more guard rails, anthropic can adjust it, whereas with open weights it's too late. It definitely sounds like the kind of thing that ends the world in B sci-fi thrillers.
- TurdF3rguson 4mo agoCurious if I'm being downvoted by people who think there will never be a model with dangerous capabilities? Or people who think when it happens we shouldn't add guard rails?