8 ms·
Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
- aarondong 4mo agoBefore getting too excited, take a look at the intelligence vs cost matrix: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task#price-cost https://artificialanalysis.ai/models?intelligence-index-toke...
- midnightbobarun 4mo ago5.6 Sol (max) being cheaper than all of these is wild, considering how good the output is too
- deleted 4mo ago[deleted]
- Schiendelman 4mo agoThis must be on API costs, not counting the $100/200 tiers, right?
- anuramat 4mo agoyes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens
- Schiendelman 4mo agoMore than that using Fable!
- giancarlostoro 4mo agoProbably because they made ASICs to run inference for less.
- brookst 4mo agoAre those actually deployed at scale yet?
- brcmthrowaway 4mo agoYes.
- wmf 4mo agoI hate to disagree with Broadcom Throwaway himself but it's unlikely that the OpenAI Jalapeno ASIC has been deployed yet. It takes 6-12 months to test, develop software, ramp production, etc.
- nijave 4mo agoI think on swebench verified luna was only like 3% points lower for 1/5 the cost Like 96% vs 93% or something
- mdgld 4mo agoYeah, sol is impressive but IMO Luna is the real standout (and terra is the laggard of the group) for performance/cost
- twotwotwo 4mo agoThere is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning. DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna max scores a smidgen lower with even more turns. Smaller models relying on lots reasoning may "scale down" better on easier tasks, because unlike size, reasoning effort is dynamic: the model can see the task looks easy and stop. On DeepSWE, the cost curves for the three 5.6 models are almost on top of each other, but on FrontierCode Extended, the version of FrontierCode with the most everyday tasks in the mix, there's a spread of costs at the ~55% level. The recent Laguna S 2.1 model (118B, 8B active) puts up surprising coding numbers for its size, and the lab behind it specifically credits its "way of working (persistence, verification, willingness to backtrack)". Some other open models that folks report getting good mileage out of seem to get there partly by throwing a lot of reasoning at the problem. There is a little bit of a question, if some models rely on getting it wrong a bit more at first and external checks catching the problems, of whether they're also more frequently getting things wrong they can't self-verify (say, quality of UI or API design) and then it falls to the human to find it. Still, getting the results they're getting at all is neat. Some benchmarks historically favored reporting only on the max variants, maybe because they want to show the frontier? but that is not always what you need for practical decision. (AA has the full effort sweep for Opus 5 and Sol/Luna/Terra at least.) And at least FrontierCode finds Opus 5 taking a hit in performance above 'medium'. I am not trying to pick a winner here. I'm probably not going to use tiny models on max for everything, but I think it's cool that you can get so much more out of a small model by amping up reasoning, tool use, and persistence.
- impulser_ 4mo agoIt shouldn't be surprising OpenAI does have the most compute out of all the major labs. The only reason why Anthropic models are expensive is they are the most in demand models in the world and Anthropic is fighting for compute. The only way to you limit demand for your model is increasing API pricing this is also why Anthropic probably has great margin and probably is profitable compared to OpenAI.
- scrlk 4mo agoNot just compute for OAI, GPT-5.6 is more token efficient across the board vs the Anthropic equivalents: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task&cost=intelligence-vs-cost-per-task&total-cost=intelligence-vs-total-cost#token-use https://artificialanalysis.ai/models?intelligence-index-toke... No wonder why Tibo can afford to hit the reset button liberally.
- charcircuit 4mo agoI also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).
- wmf 4mo ago"Price fixing" isn't the correct term here but yes, it's very common to have the same price across different retailers/resellers.
- charcircuit 4mo agoThere is a difference between the market discovering a price and a bunch of retailers/resellers entering an agreement to sell at a specific price.
- vikramkr 4mo agoI doubt there's any sort of criminal behavior there - the model is anthropic's up and anthropic probably charges a very expensive license fee that's the same for all of them, and their cogs on compute aren't going to be wildly different, so the main drivers of the cost are roughly the same and they're all offering customers the same end product so the prices would likely also be similar in the end
- eli 4mo agoMax is lot of extra reasoning. I wonder how many fewer tasks it solves on high. I bet that costs quite a lot less.
- emmp 4mo agoIndeed, you can filter the graphs to see these the values for alternative reasoning settings of the models. Opus 5 High reasoning scored 59 on the index (exactly the same as GPT 5.6 Sol Max), and costs $1.06 per task (vs $1.04 Sol Max). So these seem essentially equivalent on both metrics.
- CHUNK_CHUNK 4mo ago[flagged]
- fireant 4mo agoThat index really needs harder tasks so that it's not just a benchmark of what model is cheapest
- claude-ai 4mo agoOn my end, Opus 5 is Haiku level vs. Opus 4.8 (good) and Fable (superb). Gets confused by permission prompts, cannot debug a failing test it caused (Opus 4.8 got it right after, without tens of rounds "thinking").
- reilly3000 4mo agoAre you using Claude Code/CoWork or an API client? I’m curious if it has different training that makes it more effective with specific instructions/ tool calling methods that are only implemented in official harnesses.
- pixelesque 4mo agoI'm curious about this too, and it's difficult to get any information about this given everyone has different setups, workflows and use-cases. I bizarrely had Opus 4.8 this week (in pi.dev within a podman container, using openrouter) start installing various python packages (and uv!) within the environment (not as root) when I asked it to code review some fairly basic Rust .rs files that were generally stand-alone (it did very nicely work out and write some stubs for them to build them and work out how they worked). It only gave up with the weird Python installing stuff when it discovered one of the Python packages needed Tensorflow. It seems pretty focused and persistent in continuing its initial approach, and I'm wondering if I need to alter some instructions / initial prompts to rein it in a bit...
- firasd 4mo agoVery interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Max), Grok 4.6 (high), Gemini 3.6 Flash, GPT 5.6 Sol (Max) I've thought for a while that Gemini 3.x has 'big model smell'
- mchusma 4mo agoGemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.
- andriy_koval 4mo agothey likely tune their models for areas where they have their money: search, ads, youtube, etc.
- alex43578 4mo agoI wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).
- andriy_koval 4mo agothere were news that google uses claude internally, and also other news that Apple has its own claude tuned on internal data, maybe google has the same..
- victor106 4mo ago> knowledge tasks Like what?
- sggyamg 4mo agoIt's new, normal.
- andy99 4mo ago#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id checking - which I have not experienced personally), it’s not worth whatever slight benchmaxxing they did for the latest release.
- afavour 4mo agoWhat are you asking that you’re so regularly running into censorship?
- icedrift 4mo agoIf you even broach language related to biology you’ll get rerouted. I was presenting data in a grid and referred to a grid cell, Fable saw the word “cell” and safeguards kicked in
- jefftk 4mo agoI thought we were talking about Opus 5, the model Fable now falls back to?
- eterm 4mo agoThis thread is full of people talking confidently about their experience with a model released just hours before. Either that or everyone is indeed talking across each other and talking about different things.
- bigbuppo 4mo agoI think the big take away is that Anthropic's products are hot garbage.
- 4mo ago
- chmod775 4mo agoThe more interesting finding is that it's still the second most expensive model (after Fable 5) by a long shot. At least two models (GPT-5.6, Kimi K3) match its score (~1-2% diff) for half the cost.
- ricardobeat 4mo agoThe chart shows max effort, used mostly by price-insensitive enterprise users. At medium effort it drops to almost half K3’s cost, and is probably sufficient for 95% of coding tasks.
- stingraycharles 4mo agoYeah if there’s one thing that people should really understand it’s that it’s cheaper to have smarter models with less thinking than cheaper models with more thinking.
- andriy_koval 4mo agoOpus medium = Sol high = 56, but still 25% more expensive
- Bolwin 4mo agoFor a fair comparison, you should compare to K3 (which AA has not tested yet unfortunately) and GPT 5.6 Sol also on medium or the closest equivalent
- krzyk 4mo ago
- hoppp 4mo agoI didn't like it as much as fable. The coding style was a bit different and it way overbuilt the thing I asked from it.
- vehemenz 4mo agoIt’s crazy that people feel confident making judgments like these when the model’s been out for only a few hours.
- CommanderData 4mo agoIt's a pretty easy spot if you're already using an older model daily.
- jrs100000 4mo agoIts even crazier that people are sitting here trying to calculate intelligence per dollar from metrics. At least first impressions have more basis in real performance.
- cbg0 4mo agoIt overthinks quite a bit above medium effort, try using that.
- zormino 4mo agoI'd be curious to see the results, especially with some models having 1.5m and 2m context sizes, if the first 75% of the context was filled with unrelated info.
- theplumber 4mo agoWhy do I find it dummer/even more superficial than opus 4.8? It just continued a session and I had to stop it because it become obviously “lost”
- deleted 4mo ago[deleted]
- king_phil 4mo agoMassive performance degradation is expected when continuing a session with a different model
- protimewaster 4mo agoIs it? Is there a paper that covers this? I would've thought it should be mostly seamless, since it's being fed the entire conversation all along anyway.
- nu11ptr 4mo agoI don't have a horse in this race, but to me this makes GPT-5.6 Sol Max look better. It is about half the cost for nearly the exact same performance. It just goes to show how expensive Fable really is when Opus 5 is still this expensive relative to GPT 5.6.
- mdgld 4mo agoMake sure you’re comparing opus high to sol max. That’s where the comparison makes sense
- didibus 4mo agoWhat's interesting is this: The top AI models by Intelligence Index are: 1. Claude Opus 5 (Adaptive Reasoning, Max Effort) (61), 2. Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) (60), 3. Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (60), 4. GPT-5.6 Sol (max) (59), and 5. Claude Opus 5 (Adaptive Reasoning, High Effort) (59). Which means Opus5 at Xhigh is still smarter than Sol at max, and Opus5 at High is equal to Sol at max. That would make Opus5 High same as Sol max, and now I wonder what the price and speed difference between those is?
- theplumber 4mo agoOpus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)
- endorphine 4mo ago"him"? Have we reached that dystopia level?
- squigz 4mo agoI've been seeing a lot more anthropomorphization of these models on HN lately and it's alarming.
- perching_aix 4mo agoIt's a male name, and gendered pronouns can be hard for foreign speakers at times, irrespective of proficiency level. I wonder if you're overthinking this?
- xlii 4mo agoNot all HN visitors are native English speakers and in some languages "it" doesn't construct well with verbs, thus thought frameworks forms through usage of him/her. Nothing more to see I suppose.
- kristopolous 4mo agoI posted this before but I have a really simple shell tool to keep up with these charts over at https://github.com/day50-dev/aa-eval-email https://github.com/day50-dev/aa-eval-email This also works $ curl day50.dev/art-analysis.sh | bash Artificial analysis knows about my tool and I'm working with them on getting their API improved.
- zuzululu 4mo agoi used for several hours now and my verdict is that its no better or worse than sol its surprisingly bad at UI which is unexpected its also lacking in depth vs sol 5.6 which goes above and beyond (which in itself is also an issue at times)
- XCSme 4mo agoTwice the cost for 4% more intelligence, is it worth it?
- zkmon 4mo agoI think a more useful metric would be intelligence per dollar spent.
- luciana1u 4mo ago[flagged]
- anigbrowl 4mo agoHonestly, who the fuck cares? These leaderboards are meaningless for brand new models. If we were looking at longitudinal data collected over the course of a year or even a quarter or month, this would have some value. Brand new model from established provider shoots to top of charts? This means nothing more than an already famous band briefly topping the charts with their latest song. It baffles me that intelligent people deploying AI think momentary popularity is a meaningful signal. It's just twitch reactions * FOMO.
- jsnell 4mo agoUhh... This is not a popularity chart. It is an aggregate of benchmarks. It baffles me that somebody would write something that aggressive from that deep a level of confusion.
- deleted 4mo ago[deleted]
- NamlchakKhandro 4mo agoThis company sniffs it's own farts too much tbh
- thimbleberry 4mo ago[dead]
- entity002 4mo agoI like how Opus 5 doesn't re explain EVERYTHING to me like 4.8 did. GPT 5.6 SOL reasons WAY too hard over nothing, and Opus 5 is an amazing mode. Way to go anthropic
- ModernMech 4mo agoYes, I didn't appreciate this because I was giving Sol a brief to implement and it was doing very well. So then I just told it to do its thing without a brief and it went for 2.5 hours and used 30% of my week. I tried the same task with the brief and Sol went for 30 minutes and used 2% of my week. Compared the two and the 30 minute brief-based Sol output was much better factored, shorter, validated better, scoped better, and of course cheaper. Left to its own devices, Sol goes out of control. Now I ask Sol to write the brief and Terra to implement it, works pretty well and overall usage is down.
- codewiththiha 4mo agoI can't wait for open-source models to compete against this!
- meander_water 4mo agoThe funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model. A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task. For e.g. you might use Fable for UI design, Sol for systems design backend work and Kimi K3 for exploit development. The only purpose these metrics serve is bragging rights for the model companies.
- intothemild 4mo agoTheres value in some of AAs charts, like cost per job, and how often it hallucinated.. But I agree, wrapping that up into a single result.. you lose all the nuance, it's just bragging rights.
- andai 4mo agoA year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
- mejutoco 4mo agois there a benchmark that uses prices or speed as one of the axis, in addition to accuracy? Best could mean different things to different people.
- artemisart 4mo agoYes same website https://artificialanalysis.ai/models#intelligence-comparison-tabs https://artificialanalysis.ai/models#intelligence-comparison... but they don't have graphs for the individual benchmarks sadly.
- andai 4mo ago
- blfr 4mo agoSo if you're in Google leadership, you sleep in the office, right? Not merely because you have a ton of work but also because you're deeply ashamed to be seen in public.
- stingraycharles 4mo agoI don’t think Google cares about being the most intelligence AI as much as it cares about monetizing it with all its products, which requires speed. Google has long said that this is what it cares about most, the fastest at giving the correct answer to questions.
- mirekrusin 4mo agoThey can sleep just fine being the only player in town actually not loosing subsidized money.
- asdfasgasdgasdg 4mo agoThey've also been way behind before, and caught up to being only a little behind. We'll see how things shake out. We're deep in the present but who knows how things will look a year or two in the future.
- frogpelt 4mo agoThey are tied for first using the Google-proof question and answer benchmark: https://artificialanalysis.ai/evaluations/gpqa-diamond https://artificialanalysis.ai/evaluations/gpqa-diamond Maybe that's their only goal?
- protimewaster 4mo agoGemini models are at or near the top in several categories, though, so I'm not sure the takeaway is that they're shamefully far behind.
- scarmig 4mo agoIf you're Demis, at least, you sleep fine because you were personally an early investor in Anthropic.
- nekusar 4mo ago"When a measure becomes a target, it ceases to be a good measure" Goodhart's law.
- luxuryballs 4mo agoAnthropic has imo underrated marketing and positioning skills, mythos/fable hype/fear being the most obvious indicator but even the way they almost haphazardly position their models with no intentional cohesion, people see model names and numbers, it's easy to think of them as more intentionally accurate like how cars make S models or AMG, but then the performance and surprises surpass the prior expectation that was set by previous models, rather than having it be more obvious, suddenly the Anthropic Camry will outperform their Corvette without any fanfare.
- drob518 4mo agoNew respect for Meta Muse Spark. It seems to sit at a lot of sweet spots in the leader board. It’s not the best at anything in particular, but it balances cost and performance quite well. I’m also curious where Poolside Laguna S would sit; it’s not included. I’m personally very interested in cost effective models that still perform well.
- fHr 4mo agometrics gooners are pretty regarded
- fnord77 4mo agoFor nearly twice the price, you get 1 tick higher on some intelligence index than 5.6 Sol
- antrichards 4mo ago[flagged]
- kimjune01 4mo agoanyone else notice that the topline numbers are effort xhigh? anybody actually use the models at those levels?
- smithdamian 3mo ago[flagged]