3 ms·

DeepSWE: A contamination-free benchmark for long-horizon coding agents

67 points by ammar_x 4mo ago

ammar_x 4mo ago
https://x.com/serenaa_ge/status/2059308400866111692 https://x.com/serenaa_ge/status/2059308400866111692
dnnssl2 4mo ago
70% at launch seems pretty saturated, why ship a benchmark frontier models are about to top out on?
- charleyslee 4mo ago
  [flagged]
- vanuatu 4mo ago
  sell data for them to hillclimb :)
charleyslee 4mo ago
tysm for posting this! i'm charley, cofounder of datacurve, we created this benchmark and my team and i are here to answer any q's.
- davidshepherd7 4mo ago
  Did you try Opus-4.7 on a lower reasoning level? Looks like on `max` it's using far more tokens than the other frontier models.
- ammar_x 4mo ago
  Absolutely! We need new and better benchmarks like this. I have a question: why not use the maximum available reasoning on each LLM? For example, I see that Opus 4.7 at `max` reasoning but Sonnet 4.6 at `high`. Wouldn't it be a fairer comparison if all were at max?
toastmaster11 4mo ago
What happened that placed Opus 4.6 on max reasoning below Sonnet 4.6 on a lowered reasoning level?
vanuatu 4mo ago
This benchmark matches my experience with GPT (I occasionally go back to Claude when I run into limits and frequently run into forgotten requirements and reward hacking) I do have two questions / critiques: - The verifier doesn't seem to check for code quality / maintainability, which I would posit is one of the major qualms with SOTA coding models i.e. they lack code 'taste'. Ofc this is a difficult problem to solve at scale, but wanted to point that out nonetheless - This almost feels written like a critique on SWE Bench Pro. Hopefully they fix the issues with that benchmark!
- vanuatu 4mo ago
  Out of curiosity, I examined the worst task: https://deepswe.datacurve.ai/data/trials/quill-shared-toolbar-focus__vtVq9st https://deepswe.datacurve.ai/data/trials/quill-shared-toolba... It seems like GPT here is failing due to an environment issue of connecting to chromium, even though its local unit tests passed. All the models failed 4/4 and checking Opus it ran into the same problem I checked some other tasks and they seemed legit, although in general the prompts seem somewhat contrived vs. what a typical user would ask their coding agent (such is the difficulty of benchmark construction)
  - flakiness 4mo ago
    > Prompts are shorter than SWE-Bench Pro's but still longer than how developers actually message agents. Behavioral verification needs some minimum specificity to know what surface to test against, which puts a floor on how terse a prompt can be before the test becomes ambiguous. It'll be tricky to automate the verification with a vague prompt. In other words, the SWE's job these days is to be a intelligent verifier.
JacobAsmuth 4mo ago
I wonder why they didn't test Gemini 3.5 Flash (High).
- charleyslee 4mo ago
  in small scale testing we found high effort on gemini 3.5 flash caused it to over think, generating large amounts of tokens without a substantive improve in performance.
  - AM1010101 4mo ago
    Is that not worth running the benchmark on to prove or disprove this anyway? This would send a strong signal to google to get their act together and save me from wasting tokens selecting high. Out of curiosity how are benchmark runs generally funded? It would obviously be great to test them all on all reasoning levels and in and out of their native harnesses. Maybe even in pi / opencode / cursor but I get this would get prohibitively expensive unless you have funding or free tokens. Thanks for your efforts thus far. Looking forward to seeing more.
gertlabs 4mo ago
While this benchmark has interesting results, the "Contamination free" label only works for the initial release of the benchmark. It still has the same fundamental design issues of any other benchmark-- there's a single correct answer for tasks. It looks to be largely saturated upon release. What they did well: normalizing the harness to mini-swe-agent -- models should be able to generalize to different tools at this point. When they struggle to do that (like most Google models), they're unlikely to be useful in practice. And that kind of generalization is an inherent part of intelligence. For a benchmark that scales, you need to remove the ceiling and provide environments with measurable goals that are NOT a single correct answer, and sufficiently complex evaluation criteria to scale well beyond the current frontier. We do this by running multi-agent simulations with large action spaces at https://gertlabs.com/rankings https://gertlabs.com/rankings. We're still relatively unknown in the benchmarking space, but by rotating the pool of environments and ensuring the optimal strategies in the environments themselves are affected by other agents participating in the space, we expect we'll be able to resist contamination as major labs start investing more effort to climb the leaderboard. We've already seen Chinese labs taking an interest.
- vanuatu 4mo ago
  1. your 'agentic coding' benchmarks are already saturated, with mimo #2? Cmon 2. game rl is fundamentally less useful than coding or work rl
  - gertlabs 4mo ago
    Check out the methodology section at the bottom -- we are trying to better convey this information. 1. These numbers are based on percentiles, which inherently can't be saturated. Most benchmarks operate on something like 0-100% of correct answers, so it's natural to make that assumption when you see our numbers. Perhaps we should divide by 100. We create a modified score based on percentiles against other agents, which rebalances every time we add new entries. So when a new frontier model comes out, all of the existing entries get downweighted if the new model outperforms them. And MiMo V2.5 Pro is a much stronger model than people realize. 2. Agents write code to play most of these games (accounting for ~80% of the combined bench score). There is increasing evidence that nearly identical patterns of weights emerge in different models, trained on different mediums and using different algorithms. Pattern matching and extrapolation don't care if the scenario is a 3D "game" environment or a Salesforce "work RL" environment. Examples of drawing distant connections in different domains can reward similar circuitry.
joshka 4mo ago
How does this validate non-functional aspects of the code - docs, perf, maintainability, complexity, etc?
joshka 4mo ago
This seems like N=4, which is a fairly small sample size.

67 points by ammar_x 4mo ago

ammar_x 4mo ago

https://x.com/serenaa_ge/status/2059308400866111692 https://x.com/serenaa_ge/status/2059308400866111692

dnnssl2 4mo ago

70% at launch seems pretty saturated, why ship a benchmark frontier models are about to top out on?

charleyslee 4mo ago
[flagged]
vanuatu 4mo ago
sell data for them to hillclimb :)

charleyslee 4mo ago

tysm for posting this! i'm charley, cofounder of datacurve, we created this benchmark and my team and i are here to answer any q's.

davidshepherd7 4mo ago
Did you try Opus-4.7 on a lower reasoning level? Looks like on `max` it's using far more tokens than the other frontier models.
ammar_x 4mo ago
Absolutely! We need new and better benchmarks like this. I have a question: why not use the maximum available reasoning on each LLM? For example, I see that Opus 4.7 at `max` reasoning but Sonnet 4.6 at `high`. Wouldn't it be a fairer comparison if all were at max?

toastmaster11 4mo ago

What happened that placed Opus 4.6 on max reasoning below Sonnet 4.6 on a lowered reasoning level?

vanuatu 4mo ago

This benchmark matches my experience with GPT (I occasionally go back to Claude when I run into limits and frequently run into forgotten requirements and reward hacking) I do have two questions / critiques: - The verifier doesn't seem to check for code quality / maintainability, which I would posit is one of the major qualms with SOTA coding models i.e. they lack code 'taste'. Ofc this is a difficult problem to solve at scale, but wanted to point that out nonetheless - This almost feels written like a critique on SWE Bench Pro. Hopefully they fix the issues with that benchmark!

vanuatu 4mo ago
Out of curiosity, I examined the worst task: https://deepswe.datacurve.ai/data/trials/quill-shared-toolbar-focus__vtVq9st https://deepswe.datacurve.ai/data/trials/quill-shared-toolba... It seems like GPT here is failing due to an environment issue of connecting to chromium, even though its local unit tests passed. All the models failed 4/4 and checking Opus it ran into the same problem I checked some other tasks and they seemed legit, although in general the prompts seem somewhat contrived vs. what a typical user would ask their coding agent (such is the difficulty of benchmark construction)
- flakiness 4mo ago
  > Prompts are shorter than SWE-Bench Pro's but still longer than how developers actually message agents. Behavioral verification needs some minimum specificity to know what surface to test against, which puts a floor on how terse a prompt can be before the test becomes ambiguous. It'll be tricky to automate the verification with a vague prompt. In other words, the SWE's job these days is to be a intelligent verifier.

JacobAsmuth 4mo ago

I wonder why they didn't test Gemini 3.5 Flash (High).

charleyslee 4mo ago
in small scale testing we found high effort on gemini 3.5 flash caused it to over think, generating large amounts of tokens without a substantive improve in performance.
- AM1010101 4mo ago
  Is that not worth running the benchmark on to prove or disprove this anyway? This would send a strong signal to google to get their act together and save me from wasting tokens selecting high. Out of curiosity how are benchmark runs generally funded? It would obviously be great to test them all on all reasoning levels and in and out of their native harnesses. Maybe even in pi / opencode / cursor but I get this would get prohibitively expensive unless you have funding or free tokens. Thanks for your efforts thus far. Looking forward to seeing more.

gertlabs 4mo ago

While this benchmark has interesting results, the "Contamination free" label only works for the initial release of the benchmark. It still has the same fundamental design issues of any other benchmark-- there's a single correct answer for tasks. It looks to be largely saturated upon release. What they did well: normalizing the harness to mini-swe-agent -- models should be able to generalize to different tools at this point. When they struggle to do that (like most Google models), they're unlikely to be useful in practice. And that kind of generalization is an inherent part of intelligence. For a benchmark that scales, you need to remove the ceiling and provide environments with measurable goals that are NOT a single correct answer, and sufficiently complex evaluation criteria to scale well beyond the current frontier. We do this by running multi-agent simulations with large action spaces at https://gertlabs.com/rankings https://gertlabs.com/rankings. We're still relatively unknown in the benchmarking space, but by rotating the pool of environments and ensuring the optimal strategies in the environments themselves are affected by other agents participating in the space, we expect we'll be able to resist contamination as major labs start investing more effort to climb the leaderboard. We've already seen Chinese labs taking an interest.

vanuatu 4mo ago
1. your 'agentic coding' benchmarks are already saturated, with mimo #2? Cmon 2. game rl is fundamentally less useful than coding or work rl
- gertlabs 4mo ago
  Check out the methodology section at the bottom -- we are trying to better convey this information. 1. These numbers are based on percentiles, which inherently can't be saturated. Most benchmarks operate on something like 0-100% of correct answers, so it's natural to make that assumption when you see our numbers. Perhaps we should divide by 100. We create a modified score based on percentiles against other agents, which rebalances every time we add new entries. So when a new frontier model comes out, all of the existing entries get downweighted if the new model outperforms them. And MiMo V2.5 Pro is a much stronger model than people realize. 2. Agents write code to play most of these games (accounting for ~80% of the combined bench score). There is increasing evidence that nearly identical patterns of weights emerge in different models, trained on different mediums and using different algorithms. Pattern matching and extrapolation don't care if the scenario is a 3D "game" environment or a Salesforce "work RL" environment. Examples of drawing distant connections in different domains can reward similar circuitry.

joshka 4mo ago

How does this validate non-functional aspects of the code - docs, perf, maintainability, complexity, etc?

joshka 4mo ago

This seems like N=4, which is a fairly small sample size.