4 ms·

CursorBench 3.1

171 points by handfuloflight 4mo ago

o10449366 4mo ago
I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning? I'm starting to wonder if this increased token usage is inadvertently bleeding into how Anthropic actually trains their model, especially leading up to IPO. As older models are deprecated and users are forced onto newer models, if the default is less efficient and more token expensive that directly results in higher "profit" for Anthropic in terms of the consumption their users have to tolerate - lest they jump to a competitor.
- pbowyer 4mo ago
  > I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. I keep Claude around for some specific tasks: - Linked up to Figma MCP to implement front-end stuff - Data analysis, in the "Connect AI to a data source and ask questions" way. I've tried both Opus 4.8 high and GPT 5.5 high for this and Opus is stronger because it gets the intent in the question better I used to keep it around for planning too, but the 4.8 plans have had more holes than swiss cheese.
- anon373839 4mo ago
  > I'm starting to wonder if this increased token usage is inadvertently bleeding into how Anthropic actually trains their model Related: Sonnet 5’s new tokenizer increases token usage by 30%. (https://simonwillison.net/2026/Jun/30/claude-sonnet-5/ https://simonwillison.net/2026/Jun/30/claude-sonnet-5/)
- cbg0 4mo ago
  I've had no problems like the ones you've mentioned while using Opus 4.8. It does overthink stuff with higher effort levels but that's kind of expected.
  - mwigdahl 4mo ago
    Same (including the overthinking issue).
- mrngld 4mo ago
  Now that enterprise customers are pay-as-you-go with tokens I suspect we'll see renewed interest in OpenAI and their focus on token efficiency. At least I hope so if the alternative is abandoning the tools entirely.
anilgulecha 4mo ago
is composer 2.5 that good at that pricepoint? Seems like the gemini flash playbook of trying to get most bang for the buck.
- aabdi 4mo ago
  yes, its very good.
- danfritz 4mo ago
  It's my daily driver, it's fast affordable and with a bit of guidance gets the job done. I only reach for Claud when i need to plan something big or want to have a sparring partner to fire of some ideas. I think what a lot of people don't realize is that you don't need a fronteer model for 80% of coding tasks. Composer 2.5 is often more than good enough, less token hungry and way faster
  - shockembopper 4mo ago
    I have been doing the same for quite a while now. Composer 2.5 is incredible when you’re working in the loop.
    - simondotau 4mo ago
      When you normalise for time and money, Composer 2.5 is way, way, way, way better than anything else out there. Yes it requires more babysitting, but that's a good thing.
- uf00lme 4mo ago
  It's surprising usable and cheap enough to run in 'fast' mode when vibing something quick. For simple code I find I prefer the code it writes over GLM or Gemini family.
- fumar 4mo ago
  It’s fast and affordable.
- soyin 4mo ago
tekacs 4mo ago
I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to keep Opus on max for things that need 'real validation' for a while now. And that has felt like 'the only way' to get Opus to perform even close to 5.5 xhigh. I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window. The difference is that 5.5 xhigh is extremely fast in most practical cases, both efficiently implementing _overall_, and responding very quickly with great adaptive thinking if you ask it something that it doesn't have to think about. Opus 4.8 Max will needlessly chew on everything and can take hours to implement even simple things, so I can mostly only use it for planning/review. Fable is much much better at adaptive thinking / responding quickly (although probably still worse than 5.5 xhigh), and... I think folks have said enough elsewhere about its strengths and weaknesses. Sadly still not a reliable implementor for my hard tasks though (that's still GPT's domain) – it tends to leave big, dangerous holes hiding inside implementations unless babied.
- pbowyer 4mo ago
  > I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window. Do you find that makes a difference in your work? I've been using 5.5 high/xhigh to optimize and benchmark a C codebase, and just reading the initial code virtually fills the first context window. A session will auto-compact 5-15 times, but it seems to do okay in spite of that because the task is mainly focused on the latest window each time. I think for programming the strength of GPT over Opus is winning here over the context window.
  - tekacs 4mo ago
    > I think for programming the strength of GPT over Opus is winning here over the context window. On this, absolutely! I more often use Opus for planning than for implementation. In those cases I really do need the very large context window, because the agent has to read in a bunch of my code base and a bunch of previous plan files and product context and such, to understand what we're talking about. And then I need to go back and forth with it over a really extended period: getting into a bunch of details, asking it to load how things already work so that we can discuss options for evolution of those, etc. For that kind of thing, compaction completely destroys its effectiveness because even if you try to serialize out all the decisions made in the conversation into a plan file, the agent still loses e.g. the plan files and code files that it's read in that are adding sharp edges to its understanding of the scope of what's being planned. For implementation or something like what you're describing in the vein of benchmarking, often I can get away with compaction. Although even then, if the agent needs to have a lot "loaded" into its head, to implement something very, very subtle, complex or far-reaching, in those cases it can be really detrimental if it compacts.
mdasen 4mo ago
I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents https://artificialanalysis.ai/agents/coding-agents. You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8 max gets 56, and Cursor 2.5 gets 16. I don't doubt that Cursor works well for some people. It's beating DeepSeek v4 Pro in the DeepSWE benchmark and that's a very capable model. But I'm skeptical of the claims that it's a competitor for Opus 4.8 and GPT-5.5. It just seems convenient that their model does so well on their own benchmark while third party benchmarks have it far behind. Maybe it's a really great benchmark and a better measure than third party ones - I'd love for a cheap model to do as well as the expensive ones.
- famouswaffles 4mo ago
  Cursor sessions are pretty much what composer models are RL'd on. This bench and the training data are/should be basically the same distribution.
- datadrivenangel 4mo ago
  For lighter interactive agentic coding, where you type stuff into an IDE and a minute or three later get results back for review, composer 2.5 is honestly pretty great. The results get notably worse for larger tasks though.
  - anon7000 4mo ago
    Agreed. It’s worse than Opus of course. But Opus takes more than 10x longer to give you something to look at. I’m not kidding, I “benchmarked” a real ticket I was working on. Opus 4.7 took more than 30min. Opus 4.8 took over an hour. Composer 2.5 took 5min on the exact same prompt & local setup. My subjective review is that composer’s code was only like 10-20% worse. It still worked, it was just a bit less clean and a little more hacky. But it’s not like Opus is flawless either. At the end of the day, if it takes an hour to get to draft code I can look at and iterate on… that’s fucking impossible for me. Unless it did an excellent job. But as long as I still need to review and follow up with changes, Opus is just too slow. It’s really frustrating because it’s a lot slower than it was 6mo ago, and not noticeably better. Fable seems a step in the right direction but is $$$$
BugsJustFindMe 4mo ago
I've used both Composer 2.5 and GPT 5.5 (both in Cursor and in Codex) extensively, and their claim that Composer 2.5 is anywhere close in performance to GPT 5.5 is absolutely farcical. It's faster, but it's nowhere near as good. And given that you can only use Composer with a Cursor monthly subscription, cost comparisons are pointless since an equivalently priced OpenAI subscription gets you just as much usage of the better model.
avikaa_ 4mo ago
[flagged]
__natty__ 4mo ago
It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.
- apwheele 4mo ago
  I read these and think it is just the jagged edge. I do not doubt your personal experience, I have used Composer 2.5 (via Grok and the credits I get with my X premium account) the past month. I am not building rockets, but have been quite impressed. All the models do dumb things sometimes, it has done the work I have asked it to pretty well though and has done to me some impressive work. It is fast on Grok, for other models I have worked extensively with I think it is better than gemini 3.1 (3.5 and antigravity for me is worse than the prior gemini cli). And is comparable to Opus 4.6. (Have not used the more recent models in Claude Code.)
- bmurphy1976 4mo ago
  I'm not using Cursor at the moment, but when I did (not too long ago) my experience was similar. Plan with Opus, implement with Composer, clean up with Opus. Composer did a competent but not amazing job with a good plan. What I really liked though is it was fast! Opus could take 30 minutes to do something Composer would get done in 5-10 minutes. Of course the output wasn't perfect, but that's why I'd do a cleanup pass using Opus or Codex. It's all a balance though, constantly changing and completely dependent on the problem you're solving. I just remain flexible and adapt my process to what's working best in the moment.
  - ignoramous 4mo ago
    Interesting. If I may: What was this "clean up" pass? A code review? A code review with specialized prompt? A focused review to check for edge cases / logic errors / api misuse? Or, something else specific to the codebase? Have you settled on what the clean up pass should look like? Or, do you keep experimenting with it? In case one might not have been aware: Composer 2 was Kimi Base 2.5 post-trained (RL'd) by Cursor: https://news.ycombinator.com/item?id=48507474 https://news.ycombinator.com/item?id=48507474. Composer 2.5 might be something totally different.
verse 4mo ago
backwards X axis? is there a reason for that? it looks ridiculous
- gkbrk 4mo ago
  It looks very natural, cheaper is better after all. Performance axis going up, and cheapness axis going up match each other.
  - 0123456789ABCDE 4mo ago
    gp's argument is that cheapness is a construct, derived from the real, and natural, cost parameter which most people are naturally accustomed to interpreting as increasing from left to right. cheapness would then replace the cost label, and feel natural. alas, this is not what we have here.
- anon373839 4mo ago
  This seems to be a common choice with AI industry graphs, to give you that “upward and outward” frontier shape.
luckilydiscrete 4mo ago
insert obama medal meme
nok22kon 4mo ago
everytime a new benchmark appears, Chinese models are far lower than the level where they are supposed to be according to existing benchmarks. then after a while they recover :)
- generalpf 4mo ago
  The magic of distillation!
xrisk 4mo ago
Would like to see wall times. I feel that’s the part that annoys me most, my tasks aren’t particularly challenging I want them done fast
shadeslayer_ 4mo ago
Do these benchmarks even add any value at this point? This one is basically Cursor saying that their model is as good as the frontier ones at a fraction of the price. The independent benchmarks are probably part of training data now and the models are pattern-matching against them all the time. The final test of a model (and the harness, probably) is how good it works FOR YOU - since most of the models can pretty much do most of our tasks on a daily basis - it boils down to which one has the least friction to its usage.
bfjvibybd6cuvu6 4mo ago
No shot 2.5 is beating out 4.8
baq 4mo ago
Cursor’s model excels at Cursor’s benchmark; news at 11. The other models however are reasonably where I’d expect them to be from experience piloting all of them. Fable is outclassing everything at most things at 10x the cost, but sometimes it isn’t a choice between cheap and expensive, but expensive and possible; I’ll need to learn where that boundary is just as it was the case with other models.
xyzsparetimexyz 4mo ago
I wish all these sites would show pareto frontier graphs of cost/performance. That's the main 2 things that matter (I guess you could make it 3D with a speed param as well). https://paraplouis.github.io/llm-pareto-frontier/ https://paraplouis.github.io/llm-pareto-frontier/ is the best of these graphs I've seen but it doesn't update as frequently as I'd like.
- mrngld 4mo ago
  That site is useless though because thinking tokens (and caching) and the efficiency thereof aren't accounted for. GLM5.2 is promoted by every 50 Cent Party the PLA can muster on the internet but it falls short because of its extremely verbose thinking. Anthropic models have the same problem but starting from a much higher base of real intelligence. Which is exactly why every credible comparison now represents cost associated with completing a task, not arbitrary input and output token costs.
  - xyzsparetimexyz 4mo ago
    Do you seriously think that '五毛's are the people talking about GLM 2.5?
  - ignoramous 4mo ago
    > much higher base of real intelligence Not sure how much "real intelligence" is to be found in Mythos & Sol, but at this point, ignoring the intelligence gap, I find it totally impressive that the likes of GLM, Kimi, Qwen, MiMo hold their own at 2x to 4x less cost, and work for my use case just the same.
mi_lk 4mo ago
Cursor: Find me another benchmark where Composer 2.5 is a top 10 frontier coding model
- leerob 4mo ago
  (I work at Cursor) We score well on Terminal-Bench and SWE-bench Multilingual. DeepSWE, not so great yet, as it's more for very long-horizon tasks. We're planning to include more public benchmarks in our next model release.
tmach32 4mo ago
Why would anyone take this benchmark seriously? Cursor is obviously biased here. They can design it and its presentation however they want to tell the story they want to tell.
maxdo 4mo ago
The most interesting part is costs . Gpt 5.5 and sonnet 5 cost same amount of money as GLM 5.2 but are more capable models
kandros 4mo ago
I like Composer a lot as a general-purpose workhorse, but putting it over gpt5.5 medium makes the whole graph lose trust to me, asme witg GLM so low
andai 4mo ago
Interesting that Opus 4.7 does better than 4.8. Too bad they didn't test 4.6, too. I witnessed a man here mocked yesterday for insisting it was better than its successors! Although, the benchies are always tricksy ... On DeepSWE, GPT-5.5 beats Opus-4.8, by a fair margin, but on FrontierCode, the situation is the other way around. The only benchmark you can trust is your actual workload!
sisve 4mo ago
If I understand the graph correctly; Fable is using less tokens to achive that same tasks compared to sonet and opus. If so that is a good thing. It feels like we for a while there was spitting out tokens to get a better result. If the model themselves are getting better without generating more tokens that feels like a real win. Q1: Why is number of steps relevant in this graph? What does it tell us? Q2: and why have they flipped the horizontal graph so that 0 is to the right and not at origo? Is that some kind of new smart thing? can't say i have seen it before
Losenok 4mo ago
Very skeptical about the composer accuracy. I have been using it for 6 months now and it is very fast, especially compared to anthropic models, but the result it produces, especially with more difficult tasks is very shallow. It feels like it just finds the cheapest way to deliver the task.
karlmush 4mo ago
AI is getting expensive.
- ElijahLynn 4mo ago
  I see what you're saying. And it's still crazy valuable for the price. Compared with how much it can do in such little time, it's still far less than even a junior engineer.
richkuo 4mo ago
Composer 2.5 is really effective at some tasks, but doesn't do as well on higher complexity tasks from my own experience. With that being said, Cursor Bench is nonetheless a fantastic gauge of LLM quality and performance. The most interesting outcome of Cursor Bench is Fable 5 to GPT 5.5, its almost a perfect continuous line. As of the report, Fable to GPT might be the new standard of agentic programming/building I did my own rounds of tests for a lot of models. I made multiple rounds of comparisons for issues in my open source project using Fable 5, GPT 5.5, Opus 4.8, and Composer 2.5 Made them work on various complexity issues Here are the reports, i recommend using a Sonnet 5 or better model to summarize them because its quite a bit of information to consume across all the various tests: https://www.richkuo.com/#llm-battles https://www.richkuo.com/#llm-battles My takeaway from Composer 2.5 is that its better to use Opus or GPT for planning, and then Composer to build, use Opus/GPT to pr review, and have Composer fix findings, and loop that process. Incredibly token efficient, fast, and gets the same quality as if you worked the whole thing with Fable/Opus/GPT.

171 points by handfuloflight 4mo ago

I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning? I'm starting to wonder if this increased token usage is inadvertently bleeding into how Anthropic actually trains their model, especially leading up to IPO. As older models are deprecated and users are forced onto newer models, if the default is less efficient and more token expensive that directly results in higher "profit" for Anthropic in terms of the consumption their users have to tolerate - lest they jump to a competitor.

pbowyer 4mo ago
> I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks. I keep Claude around for some specific tasks: - Linked up to Figma MCP to implement front-end stuff - Data analysis, in the "Connect AI to a data source and ask questions" way. I've tried both Opus 4.8 high and GPT 5.5 high for this and Opus is stronger because it gets the intent in the question better I used to keep it around for planning too, but the 4.8 plans have had more holes than swiss cheese.
anon373839 4mo ago
> I'm starting to wonder if this increased token usage is inadvertently bleeding into how Anthropic actually trains their model Related: Sonnet 5’s new tokenizer increases token usage by 30%. (https://simonwillison.net/2026/Jun/30/claude-sonnet-5/ https://simonwillison.net/2026/Jun/30/claude-sonnet-5/)
cbg0 4mo ago
I've had no problems like the ones you've mentioned while using Opus 4.8. It does overthink stuff with higher effort levels but that's kind of expected.
- mwigdahl 4mo ago
  Same (including the overthinking issue).
mrngld 4mo ago
Now that enterprise customers are pay-as-you-go with tokens I suspect we'll see renewed interest in OpenAI and their focus on token efficiency. At least I hope so if the alternative is abandoning the tools entirely.

anilgulecha 4mo ago

is composer 2.5 that good at that pricepoint? Seems like the gemini flash playbook of trying to get most bang for the buck.

aabdi 4mo ago
yes, its very good.
danfritz 4mo ago
It's my daily driver, it's fast affordable and with a bit of guidance gets the job done. I only reach for Claud when i need to plan something big or want to have a sparring partner to fire of some ideas. I think what a lot of people don't realize is that you don't need a fronteer model for 80% of coding tasks. Composer 2.5 is often more than good enough, less token hungry and way faster
- shockembopper 4mo ago
  I have been doing the same for quite a while now. Composer 2.5 is incredible when you’re working in the loop.
  - simondotau 4mo ago
    When you normalise for time and money, Composer 2.5 is way, way, way, way better than anything else out there. Yes it requires more babysitting, but that's a good thing.
uf00lme 4mo ago
It's surprising usable and cheap enough to run in 'fast' mode when vibing something quick. For simple code I find I prefer the code it writes over GLM or Gemini family.
fumar 4mo ago
It’s fast and affordable.
soyin 4mo ago

tekacs 4mo ago

I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO. Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to keep Opus on max for things that need 'real validation' for a while now. And that has felt like 'the only way' to get Opus to perform even close to 5.5 xhigh. I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window. The difference is that 5.5 xhigh is extremely fast in most practical cases, both efficiently implementing _overall_, and responding very quickly with great adaptive thinking if you ask it something that it doesn't have to think about. Opus 4.8 Max will needlessly chew on everything and can take hours to implement even simple things, so I can mostly only use it for planning/review. Fable is much much better at adaptive thinking / responding quickly (although probably still worse than 5.5 xhigh), and... I think folks have said enough elsewhere about its strengths and weaknesses. Sadly still not a reliable implementor for my hard tasks though (that's still GPT's domain) – it tends to leave big, dangerous holes hiding inside implementations unless babied.

pbowyer 4mo ago
> I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window. Do you find that makes a difference in your work? I've been using 5.5 high/xhigh to optimize and benchmark a C codebase, and just reading the initial code virtually fills the first context window. A session will auto-compact 5-15 times, but it seems to do okay in spite of that because the task is mainly focused on the latest window each time. I think for programming the strength of GPT over Opus is winning here over the context window.
- tekacs 4mo ago
  > I think for programming the strength of GPT over Opus is winning here over the context window. On this, absolutely! I more often use Opus for planning than for implementation. In those cases I really do need the very large context window, because the agent has to read in a bunch of my code base and a bunch of previous plan files and product context and such, to understand what we're talking about. And then I need to go back and forth with it over a really extended period: getting into a bunch of details, asking it to load how things already work so that we can discuss options for evolution of those, etc. For that kind of thing, compaction completely destroys its effectiveness because even if you try to serialize out all the decisions made in the conversation into a plan file, the agent still loses e.g. the plan files and code files that it's read in that are adding sharp edges to its understanding of the scope of what's being planned. For implementation or something like what you're describing in the vein of benchmarking, often I can get away with compaction. Although even then, if the agent needs to have a lot "loaded" into its head, to implement something very, very subtle, complex or far-reaching, in those cases it can be really detrimental if it compacts.

mdasen 4mo ago

I'm a bit skeptical. Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price. Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents https://artificialanalysis.ai/agents/coding-agents. You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8 max gets 56, and Cursor 2.5 gets 16. I don't doubt that Cursor works well for some people. It's beating DeepSeek v4 Pro in the DeepSWE benchmark and that's a very capable model. But I'm skeptical of the claims that it's a competitor for Opus 4.8 and GPT-5.5. It just seems convenient that their model does so well on their own benchmark while third party benchmarks have it far behind. Maybe it's a really great benchmark and a better measure than third party ones - I'd love for a cheap model to do as well as the expensive ones.

famouswaffles 4mo ago
Cursor sessions are pretty much what composer models are RL'd on. This bench and the training data are/should be basically the same distribution.
datadrivenangel 4mo ago
For lighter interactive agentic coding, where you type stuff into an IDE and a minute or three later get results back for review, composer 2.5 is honestly pretty great. The results get notably worse for larger tasks though.
- anon7000 4mo ago
  Agreed. It’s worse than Opus of course. But Opus takes more than 10x longer to give you something to look at. I’m not kidding, I “benchmarked” a real ticket I was working on. Opus 4.7 took more than 30min. Opus 4.8 took over an hour. Composer 2.5 took 5min on the exact same prompt & local setup. My subjective review is that composer’s code was only like 10-20% worse. It still worked, it was just a bit less clean and a little more hacky. But it’s not like Opus is flawless either. At the end of the day, if it takes an hour to get to draft code I can look at and iterate on… that’s fucking impossible for me. Unless it did an excellent job. But as long as I still need to review and follow up with changes, Opus is just too slow. It’s really frustrating because it’s a lot slower than it was 6mo ago, and not noticeably better. Fable seems a step in the right direction but is $$$$

BugsJustFindMe 4mo ago

I've used both Composer 2.5 and GPT 5.5 (both in Cursor and in Codex) extensively, and their claim that Composer 2.5 is anywhere close in performance to GPT 5.5 is absolutely farcical. It's faster, but it's nowhere near as good. And given that you can only use Composer with a Cursor monthly subscription, cost comparisons are pointless since an equivalently priced OpenAI subscription gets you just as much usage of the better model.

avikaa_ 4mo ago

[flagged]

__natty__ 4mo ago

It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.

apwheele 4mo ago
I read these and think it is just the jagged edge. I do not doubt your personal experience, I have used Composer 2.5 (via Grok and the credits I get with my X premium account) the past month. I am not building rockets, but have been quite impressed. All the models do dumb things sometimes, it has done the work I have asked it to pretty well though and has done to me some impressive work. It is fast on Grok, for other models I have worked extensively with I think it is better than gemini 3.1 (3.5 and antigravity for me is worse than the prior gemini cli). And is comparable to Opus 4.6. (Have not used the more recent models in Claude Code.)
bmurphy1976 4mo ago
I'm not using Cursor at the moment, but when I did (not too long ago) my experience was similar. Plan with Opus, implement with Composer, clean up with Opus. Composer did a competent but not amazing job with a good plan. What I really liked though is it was fast! Opus could take 30 minutes to do something Composer would get done in 5-10 minutes. Of course the output wasn't perfect, but that's why I'd do a cleanup pass using Opus or Codex. It's all a balance though, constantly changing and completely dependent on the problem you're solving. I just remain flexible and adapt my process to what's working best in the moment.
- ignoramous 4mo ago
  Interesting. If I may: What was this "clean up" pass? A code review? A code review with specialized prompt? A focused review to check for edge cases / logic errors / api misuse? Or, something else specific to the codebase? Have you settled on what the clean up pass should look like? Or, do you keep experimenting with it? In case one might not have been aware: Composer 2 was Kimi Base 2.5 post-trained (RL'd) by Cursor: https://news.ycombinator.com/item?id=48507474 https://news.ycombinator.com/item?id=48507474. Composer 2.5 might be something totally different.

verse 4mo ago

backwards X axis? is there a reason for that? it looks ridiculous

gkbrk 4mo ago
It looks very natural, cheaper is better after all. Performance axis going up, and cheapness axis going up match each other.
- 0123456789ABCDE 4mo ago
  gp's argument is that cheapness is a construct, derived from the real, and natural, cost parameter which most people are naturally accustomed to interpreting as increasing from left to right. cheapness would then replace the cost label, and feel natural. alas, this is not what we have here.
anon373839 4mo ago
This seems to be a common choice with AI industry graphs, to give you that “upward and outward” frontier shape.

luckilydiscrete 4mo ago

insert obama medal meme

nok22kon 4mo ago

everytime a new benchmark appears, Chinese models are far lower than the level where they are supposed to be according to existing benchmarks. then after a while they recover :)

generalpf 4mo ago
The magic of distillation!

xrisk 4mo ago

Would like to see wall times. I feel that’s the part that annoys me most, my tasks aren’t particularly challenging I want them done fast

shadeslayer_ 4mo ago

Do these benchmarks even add any value at this point? This one is basically Cursor saying that their model is as good as the frontier ones at a fraction of the price. The independent benchmarks are probably part of training data now and the models are pattern-matching against them all the time. The final test of a model (and the harness, probably) is how good it works FOR YOU - since most of the models can pretty much do most of our tasks on a daily basis - it boils down to which one has the least friction to its usage.

bfjvibybd6cuvu6 4mo ago

No shot 2.5 is beating out 4.8

baq 4mo ago

Cursor’s model excels at Cursor’s benchmark; news at 11. The other models however are reasonably where I’d expect them to be from experience piloting all of them. Fable is outclassing everything at most things at 10x the cost, but sometimes it isn’t a choice between cheap and expensive, but expensive and possible; I’ll need to learn where that boundary is just as it was the case with other models.

xyzsparetimexyz 4mo ago

I wish all these sites would show pareto frontier graphs of cost/performance. That's the main 2 things that matter (I guess you could make it 3D with a speed param as well). https://paraplouis.github.io/llm-pareto-frontier/ https://paraplouis.github.io/llm-pareto-frontier/ is the best of these graphs I've seen but it doesn't update as frequently as I'd like.

mrngld 4mo ago
That site is useless though because thinking tokens (and caching) and the efficiency thereof aren't accounted for. GLM5.2 is promoted by every 50 Cent Party the PLA can muster on the internet but it falls short because of its extremely verbose thinking. Anthropic models have the same problem but starting from a much higher base of real intelligence. Which is exactly why every credible comparison now represents cost associated with completing a task, not arbitrary input and output token costs.
- xyzsparetimexyz 4mo ago
  Do you seriously think that '五毛's are the people talking about GLM 2.5?
- ignoramous 4mo ago
  > much higher base of real intelligence Not sure how much "real intelligence" is to be found in Mythos & Sol, but at this point, ignoring the intelligence gap, I find it totally impressive that the likes of GLM, Kimi, Qwen, MiMo hold their own at 2x to 4x less cost, and work for my use case just the same.

mi_lk 4mo ago

Cursor: Find me another benchmark where Composer 2.5 is a top 10 frontier coding model

leerob 4mo ago
(I work at Cursor) We score well on Terminal-Bench and SWE-bench Multilingual. DeepSWE, not so great yet, as it's more for very long-horizon tasks. We're planning to include more public benchmarks in our next model release.

tmach32 4mo ago

Why would anyone take this benchmark seriously? Cursor is obviously biased here. They can design it and its presentation however they want to tell the story they want to tell.

maxdo 4mo ago

The most interesting part is costs . Gpt 5.5 and sonnet 5 cost same amount of money as GLM 5.2 but are more capable models

kandros 4mo ago

I like Composer a lot as a general-purpose workhorse, but putting it over gpt5.5 medium makes the whole graph lose trust to me, asme witg GLM so low

andai 4mo ago

Interesting that Opus 4.7 does better than 4.8. Too bad they didn't test 4.6, too. I witnessed a man here mocked yesterday for insisting it was better than its successors! Although, the benchies are always tricksy ... On DeepSWE, GPT-5.5 beats Opus-4.8, by a fair margin, but on FrontierCode, the situation is the other way around. The only benchmark you can trust is your actual workload!

sisve 4mo ago

If I understand the graph correctly; Fable is using less tokens to achive that same tasks compared to sonet and opus. If so that is a good thing. It feels like we for a while there was spitting out tokens to get a better result. If the model themselves are getting better without generating more tokens that feels like a real win. Q1: Why is number of steps relevant in this graph? What does it tell us? Q2: and why have they flipped the horizontal graph so that 0 is to the right and not at origo? Is that some kind of new smart thing? can't say i have seen it before

Losenok 4mo ago

Very skeptical about the composer accuracy. I have been using it for 6 months now and it is very fast, especially compared to anthropic models, but the result it produces, especially with more difficult tasks is very shallow. It feels like it just finds the cheapest way to deliver the task.

karlmush 4mo ago

AI is getting expensive.

ElijahLynn 4mo ago
I see what you're saying. And it's still crazy valuable for the price. Compared with how much it can do in such little time, it's still far less than even a junior engineer.

richkuo 4mo ago

Composer 2.5 is really effective at some tasks, but doesn't do as well on higher complexity tasks from my own experience. With that being said, Cursor Bench is nonetheless a fantastic gauge of LLM quality and performance. The most interesting outcome of Cursor Bench is Fable 5 to GPT 5.5, its almost a perfect continuous line. As of the report, Fable to GPT might be the new standard of agentic programming/building I did my own rounds of tests for a lot of models. I made multiple rounds of comparisons for issues in my open source project using Fable 5, GPT 5.5, Opus 4.8, and Composer 2.5 Made them work on various complexity issues Here are the reports, i recommend using a Sonnet 5 or better model to summarize them because its quite a bit of information to consume across all the various tests: https://www.richkuo.com/#llm-battles https://www.richkuo.com/#llm-battles My takeaway from Composer 2.5 is that its better to use Opus or GPT for planning, and then Composer to build, use Opus/GPT to pr review, and have Composer fix findings, and loop that process. Incredibly token efficient, fast, and gets the same quality as if you worked the whole thing with Fable/Opus/GPT.