5 ms·
Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
- aarondong 2mo agoBefore getting too excited, take a look at the intelligence vs cost matrix: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task#price-cost https://artificialanalysis.ai/models?intelligence-index-toke...
- midnightbobarun 2mo ago5.6 Sol (max) being cheaper than all of these is wild, considering how good the output is too
- deleted 2mo ago[deleted]
- Schiendelman 2mo agoThis must be on API costs, not counting the $100/200 tiers, right?
- anuramat 2mo agoyes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens
- Schiendelman 2mo agoMore than that using Fable!
- giancarlostoro 2mo agoProbably because they made ASICs to run inference for less.
- brookst 2mo agoAre those actually deployed at scale yet?
- brcmthrowaway 2mo agoYes.
- wmf 2mo agoI hate to disagree with Broadcom Throwaway himself but it's unlikely that the OpenAI Jalapeno ASIC has been deployed yet. It takes 6-12 months to test, develop software, ramp production, etc.
- nijave 2mo agoI think on swebench verified luna was only like 3% points lower for 1/5 the cost Like 96% vs 93% or something
- mdgld 2mo agoYeah, sol is impressive but IMO Luna is the real standout (and terra is the laggard of the group) for performance/cost
- twotwotwo 2mo agoThere is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning. DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna max scores a smidgen lower with even more turns. Smaller models relying on lots reasoning may "scale down" better on easier tasks, because unlike size, reasoning effort is dynamic: the model can see the task looks easy and stop. On DeepSWE, the cost curves for the three 5.6 models are almost on top of each other, but on FrontierCode Extended, the version of FrontierCode with the most everyday tasks in the mix, there's a spread of costs at the ~55% level. The recent Laguna S 2.1 model (118B, 8B active) puts up surprising coding numbers for its size, and the lab behind it specifically credits its "way of working (persistence, verification, willingness to backtrack)". Some other open models that folks report getting good mileage out of seem to get there partly by throwing a lot of reasoning at the problem. There is a little bit of a question, if some models rely on getting it wrong a bit more at first and external checks catching the problems, of whether they're also more frequently getting things wrong they can't self-verify (say, quality of UI or API design) and then it falls to the human to find it. Still, getting the results they're getting at all is neat. Some benchmarks historically favored reporting only on the max variants, maybe because they want to show the frontier? but that is not always what you need for practical decision. (AA has the full effort sweep for Opus 5 and Sol/Luna/Terra at least.) And at least FrontierCode finds Opus 5 taking a hit in performance above 'medium'. I am not trying to pick a winner here. I'm probably not going to use tiny models on max for everything, but I think it's cool that you can get so much more out of a small model by amping up reasoning, tool use, and persistence.
- impulser_ 2mo agoIt shouldn't be surprising OpenAI does have the most compute out of all the major labs. The only reason why Anthropic models are expensive is they are the most in demand models in the world and Anthropic is fighting for compute. The only way to you limit demand for your model is increasing API pricing this is also why Anthropic probably has great margin and probably is profitable compared to OpenAI.
- scrlk 2mo agoNot just compute for OAI, GPT-5.6 is more token efficient across the board vs the Anthropic equivalents: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task&cost=intelligence-vs-cost-per-task&total-cost=intelligence-vs-total-cost#token-use https://artificialanalysis.ai/models?intelligence-index-toke... No wonder why Tibo can afford to hit the reset button liberally.
- charcircuit 2mo agoI also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).
- wmf 2mo ago"Price fixing" isn't the correct term here but yes, it's very common to have the same price across different retailers/resellers.
- charcircuit 2mo agoThere is a difference between the market discovering a price and a bunch of retailers/resellers entering an agreement to sell at a specific price.
- vikramkr 2mo agoI doubt there's any sort of criminal behavior there - the model is anthropic's up and anthropic probably charges a very expensive license fee that's the same for all of them, and their cogs on compute aren't going to be wildly different, so the main drivers of the cost are roughly the same and they're all offering customers the same end product so the prices would likely also be similar in the end
- eli 2mo agoMax is lot of extra reasoning. I wonder how many fewer tasks it solves on high. I bet that costs quite a lot less.
- emmp 2mo agoIndeed, you can filter the graphs to see these the values for alternative reasoning settings of the models. Opus 5 High reasoning scored 59 on the index (exactly the same as GPT 5.6 Sol Max), and costs $1.06 per task (vs $1.04 Sol Max). So these seem essentially equivalent on both metrics.
- CHUNK_CHUNK 2mo ago[flagged]
- fireant 2mo agoThat index really needs harder tasks so that it's not just a benchmark of what model is cheapest
- claude-ai 2mo agoOn my end, Opus 5 is Haiku level vs. Opus 4.8 (good) and Fable (superb). Gets confused by permission prompts, cannot debug a failing test it caused (Opus 4.8 got it right after, without tens of rounds "thinking").
- reilly3000 2mo agoAre you using Claude Code/CoWork or an API client? I’m curious if it has different training that makes it more effective with specific instructions/ tool calling methods that are only implemented in official harnesses.
- pixelesque 2mo agoI'm curious about this too, and it's difficult to get any information about this given everyone has different setups, workflows and use-cases. I bizarrely had Opus 4.8 this week (in pi.dev within a podman container, using openrouter) start installing various python packages (and uv!) within the environment (not as root) when I asked it to code review some fairly basic Rust .rs files that were generally stand-alone (it did very nicely work out and write some stubs for them to build them and work out how they worked). It only gave up with the weird Python installing stuff when it discovered one of the Python packages needed Tensorflow. It seems pretty focused and persistent in continuing its initial approach, and I'm wondering if I need to alter some instructions / initial prompts to rein it in a bit...
- firasd 2mo agoVery interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Max), Grok 4.6 (high), Gemini 3.6 Flash, GPT 5.6 Sol (Max) I've thought for a while that Gemini 3.x has 'big model smell'
- mchusma 2mo agoGemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.
- andriy_koval 2mo agothey likely tune their models for areas where they have their money: search, ads, youtube, etc.
- alex43578 2mo agoI wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).
- andriy_koval 2mo agothere were news that google uses claude internally, and also other news that Apple has its own claude tuned on internal data, maybe google has the same..
- victor106 2mo ago> knowledge tasks Like what?
- sggyamg 2mo agoIt's new, normal.
- andy99 2mo ago#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id checking - which I have not experienced personally), it’s not worth whatever slight benchmaxxing they did for the latest release.
- afavour 2mo agoWhat are you asking that you’re so regularly running into censorship?
- icedrift 2mo agoIf you even broach language related to biology you’ll get rerouted. I was presenting data in a grid and referred to a grid cell, Fable saw the word “cell” and safeguards kicked in
- jefftk 2mo agoI thought we were talking about Opus 5, the model Fable now falls back to?
- eterm 2mo agoThis thread is full of people talking confidently about their experience with a model released just hours before. Either that or everyone is indeed talking across each other and talking about different things.
- bigbuppo 2mo agoI think the big take away is that Anthropic's products are hot garbage.
- 2mo ago
- chmod775 2mo agoThe more interesting finding is that it's still the second most expensive model (after Fable 5) by a long shot. At least two models (GPT-5.6, Kimi K3) match its score (~1-2% diff) for half the cost.
- ricardobeat 2mo agoThe chart shows max effort, used mostly by price-insensitive enterprise users. At medium effort it drops to almost half K3’s cost, and is probably sufficient for 95% of coding tasks.
- stingraycharles 2mo agoYeah if there’s one thing that people should really understand it’s that it’s cheaper to have smarter models with less thinking than cheaper models with more thinking.
- andriy_koval 2mo agoOpus medium = Sol high = 56, but still 25% more expensive
- Bolwin 2mo agoFor a fair comparison, you should compare to K3 (which AA has not tested yet unfortunately) and GPT 5.6 Sol also on medium or the closest equivalent
- krzyk 2mo ago
- hoppp 2mo agoI didn't like it as much as fable. The coding style was a bit different and it way overbuilt the thing I asked from it.
- vehemenz 2mo agoIt’s crazy that people feel confident making judgments like these when the model’s been out for only a few hours.
- CommanderData 2mo agoIt's a pretty easy spot if you're already using an older model daily.
- jrs100000 2mo agoIts even crazier that people are sitting here trying to calculate intelligence per dollar from metrics. At least first impressions have more basis in real performance.
- cbg0 2mo agoIt overthinks quite a bit above medium effort, try using that.
- zormino 2mo agoI'd be curious to see the results, especially with some models having 1.5m and 2m context sizes, if the first 75% of the context was filled with unrelated info.
- theplumber 2mo agoWhy do I find it dummer/even more superficial than opus 4.8? It just continued a session and I had to stop it because it become obviously “lost”
- deleted 2mo ago[deleted]
- king_phil 2mo agoMassive performance degradation is expected when continuing a session with a different model
- protimewaster 2mo agoIs it? Is there a paper that covers this? I would've thought it should be mostly seamless, since it's being fed the entire conversation all along anyway.
- nu11ptr 2mo agoI don't have a horse in this race, but to me this makes GPT-5.6 Sol Max look better. It is about half the cost for nearly the exact same performance. It just goes to show how expensive Fable really is when Opus 5 is still this expensive relative to GPT 5.6.
- mdgld 2mo agoMake sure you’re comparing opus high to sol max. That’s where the comparison makes sense
- didibus 2mo agoWhat's interesting is this: The top AI models by Intelligence Index are: 1. Claude Opus 5 (Adaptive Reasoning, Max Effort) (61), 2. Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) (60), 3. Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (60), 4. GPT-5.6 Sol (max) (59), and 5. Claude Opus 5 (Adaptive Reasoning, High Effort) (59). Which means Opus5 at Xhigh is still smarter than Sol at max, and Opus5 at High is equal to Sol at max. That would make Opus5 High same as Sol max, and now I wonder what the price and speed difference between those is?
- theplumber 2mo agoOpus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)
- endorphine 2mo ago"him"? Have we reached that dystopia level?
- squigz 2mo agoI've been seeing a lot more anthropomorphization of these models on HN lately and it's alarming.
- perching_aix 2mo agoIt's a male name, and gendered pronouns can be hard for foreign speakers at times, irrespective of proficiency level. I wonder if you're overthinking this?
- xlii 2mo agoNot all HN visitors are native English speakers and in some languages "it" doesn't construct well with verbs, thus thought frameworks forms through usage of him/her. Nothing more to see I suppose.
- kristopolous 2mo agoI posted this before but I have a really simple shell tool to keep up with these charts over at https://github.com/day50-dev/aa-eval-email https://github.com/day50-dev/aa-eval-email This also works $ curl day50.dev/art-analysis.sh | bash Artificial analysis knows about my tool and I'm working with them on getting their API improved.
- zuzululu 2mo agoi used for several hours now and my verdict is that its no better or worse than sol its surprisingly bad at UI which is unexpected its also lacking in depth vs sol 5.6 which goes above and beyond (which in itself is also an issue at times)
- XCSme 2mo agoTwice the cost for 4% more intelligence, is it worth it?
- zkmon 2mo agoI think a more useful metric would be intelligence per dollar spent.
- luciana1u 2mo ago[flagged]
- anigbrowl 2mo agoHonestly, who the fuck cares? These leaderboards are meaningless for brand new models. If we were looking at longitudinal data collected over the course of a year or even a quarter or month, this would have some value. Brand new model from established provider shoots to top of charts? This means nothing more than an already famous band briefly topping the charts with their latest song. It baffles me that intelligent people deploying AI think momentary popularity is a meaningful signal. It's just twitch reactions * FOMO.
- jsnell 2mo agoUhh... This is not a popularity chart. It is an aggregate of benchmarks. It baffles me that somebody would write something that aggressive from that deep a level of confusion.
- deleted 2mo ago[deleted]
- NamlchakKhandro 2mo agoThis company sniffs it's own farts too much tbh
- thimbleberry 2mo ago[dead]
- entity002 2mo agoI like how Opus 5 doesn't re explain EVERYTHING to me like 4.8 did. GPT 5.6 SOL reasons WAY too hard over nothing, and Opus 5 is an amazing mode. Way to go anthropic
- ModernMech 2mo agoYes, I didn't appreciate this because I was giving Sol a brief to implement and it was doing very well. So then I just told it to do its thing without a brief and it went for 2.5 hours and used 30% of my week. I tried the same task with the brief and Sol went for 30 minutes and used 2% of my week. Compared the two and the 30 minute brief-based Sol output was much better factored, shorter, validated better, scoped better, and of course cheaper. Left to its own devices, Sol goes out of control. Now I ask Sol to write the brief and Terra to implement it, works pretty well and overall usage is down.
- codewiththiha 2mo agoI can't wait for open-source models to compete against this!
- meander_water 2mo agoThe funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model. A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task. For e.g. you might use Fable for UI design, Sol for systems design backend work and Kimi K3 for exploit development. The only purpose these metrics serve is bragging rights for the model companies.
- intothemild 2mo agoTheres value in some of AAs charts, like cost per job, and how often it hallucinated.. But I agree, wrapping that up into a single result.. you lose all the nuance, it's just bragging rights.
- andai 2mo agoA year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
- mejutoco 2mo agois there a benchmark that uses prices or speed as one of the axis, in addition to accuracy? Best could mean different things to different people.
- artemisart 2mo agoYes same website https://artificialanalysis.ai/models#intelligence-comparison-tabs https://artificialanalysis.ai/models#intelligence-comparison... but they don't have graphs for the individual benchmarks sadly.
- andai 2mo ago
- blfr 2mo agoSo if you're in Google leadership, you sleep in the office, right? Not merely because you have a ton of work but also because you're deeply ashamed to be seen in public.
- stingraycharles 2mo agoI don’t think Google cares about being the most intelligence AI as much as it cares about monetizing it with all its products, which requires speed. Google has long said that this is what it cares about most, the fastest at giving the correct answer to questions.
- mirekrusin 2mo agoThey can sleep just fine being the only player in town actually not loosing subsidized money.
- asdfasgasdgasdg 2mo agoThey've also been way behind before, and caught up to being only a little behind. We'll see how things shake out. We're deep in the present but who knows how things will look a year or two in the future.
- frogpelt 2mo agoThey are tied for first using the Google-proof question and answer benchmark: https://artificialanalysis.ai/evaluations/gpqa-diamond https://artificialanalysis.ai/evaluations/gpqa-diamond Maybe that's their only goal?
- protimewaster 2mo agoGemini models are at or near the top in several categories, though, so I'm not sure the takeaway is that they're shamefully far behind.
- scarmig 2mo agoIf you're Demis, at least, you sleep fine because you were personally an early investor in Anthropic.
- nekusar 2mo ago"When a measure becomes a target, it ceases to be a good measure" Goodhart's law.
- luxuryballs 2mo agoAnthropic has imo underrated marketing and positioning skills, mythos/fable hype/fear being the most obvious indicator but even the way they almost haphazardly position their models with no intentional cohesion, people see model names and numbers, it's easy to think of them as more intentionally accurate like how cars make S models or AMG, but then the performance and surprises surpass the prior expectation that was set by previous models, rather than having it be more obvious, suddenly the Anthropic Camry will outperform their Corvette without any fanfare.
- drob518 2mo agoNew respect for Meta Muse Spark. It seems to sit at a lot of sweet spots in the leader board. It’s not the best at anything in particular, but it balances cost and performance quite well. I’m also curious where Poolside Laguna S would sit; it’s not included. I’m personally very interested in cost effective models that still perform well.
- fHr 2mo agometrics gooners are pretty regarded
- fnord77 2mo agoFor nearly twice the price, you get 1 tick higher on some intelligence index than 5.6 Sol
- antrichards 2mo ago[flagged]
- kimjune01 2mo agoanyone else notice that the topline numbers are effort xhigh? anybody actually use the models at those levels?
- smithdamian 2mo ago[flagged]