5 ms·

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

125 points by mpweiher 8mo ago

verdverm 8mo ago
Really long-term task benchmark showing significant improvements in very recent models, while also showing really bad regression rates across the board.
- woeirua 7mo ago
  Uh, Opus 4.6 avoids introducing regressions 75% of the time?
  - verdverm 7mo ago
    So 1/4 times it does not introduce a regression. That's still pretty bad imo. If 1/4 commits introduced regressions, what would your team do? We are talking about regressions, what once worked no longer does, and should be measured in 9s
    - notduncansmith 7mo ago
      You overestimate many teams I think.
devcraft_ai 8mo ago
[flagged]
challengerVIE 8mo ago
To me using agents daily, the long term vision with maintainability in mind really makes the difference between us humans and agents, I like the idea. However evaluating long term maintainability over an average of just 500 loc changes does not sound like long term maintainability being measured here
KronisLV 8mo ago
> The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. This seems like a really cool thing to benchmark! Technically it'd be possible to take GitHub repos that the AI orgs probably already have, cross-reference the code against the issues and regressions, and train/validate on that. The dataset would need to be way bigger to get close to the likes of SWE-bench: https://www.swebench.com/original.html https://www.swebench.com/original.html "Vibe coded stuff gets hard to maintain and will end up buggy." Yeah, so make models that deal with that better, optimize for maintainability and consistency. Cool to see Claude doing decently though!
- woadwarrior01 8mo ago
  > Cool to see Claude doing decently though! The scales do seem to be tipped in its favor (cf: my other comment in this thread).
woadwarrior01 8mo ago
Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).
- aurareturn 8mo ago
  As far as I know, OpenAI did not release 5.3 Codex in their API. You can only use it with Codex CLI or app.
  - baalimago 7mo ago
    It's there, you just need to use it with the responses API. Set model field to 'gpt-5.3-codex'
- re-thc 7mo ago
  5.2 and 5.2 Codex is arguably the same gen.
  - jasonjmcghee 7mo ago
    Sure, but one is fine-tuned for what they are testing and one is not.
mentalgear 8mo ago
Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these models with added agent tooling ("harness") ? Maybe someone has tokens to burn and can run a matrix of agent tools over the top models and provide the results?
- mike_hearn 7mo ago
  It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.
  - pizlonator 7mo ago
    gpt-5.3 was not accessible via API, at least for me But it was in codex
    - mike_hearn 7mo ago
      Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than the underlying models.
      - pizlonator 7mo ago
        I am using responses API (these days I only use responses API when talking to OpenAI) I'll try again, maybe I was doing something else wrong, or something changed
50lo 8mo ago
It’d be interesting to see this compared against a human baseline — e.g., a competent engineer with a fixed time budget on the same tasks.
PunchyHamster 7mo ago
I'm sure with benchmarks like these future LLMs will be optimized to hide regressions by "fixing" test framework too
- pixl97 7mo ago
  Isn't misalignment great.
baalimago 7mo ago
Replace "Agent" with "Employee" and apply the same algorithm. Evaluate employee efficiency. Profit?
- KronisLV 7mo ago
  I'd unironically (and privately) want to do that with the code of both myself and those around me - to maybe see who I should listen more to, as well as who maybe less (ideally down to the feature level), because everyone has opinions, sometimes loud ones, but some approaches lead to a lot of churn and issues over the years.
gizmodo59 7mo ago
Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api https://developers.openai.com/api/docs/models/gpt-5.3-codex https://developers.openai.com/api/docs/models/gpt-5.3-codex IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if you are running any kind of agentic coding tasks. I see both Claude and gpt to be neck and neck in coding. Every other model+harness is definitely 3-6 months behind. Right now codex seems to be the best in terms of solving complex bugs, long running tasks, much higher limits and even speed while Claude seems to do well in front end and their cli ux seems nice! Codex app is very good though (wish it wasn’t electron as a memory hog but it’s good)
- p1esk 7mo ago
  Are you saying they did not use native harnesses like Claude Code or Codex? How did they do it then?
- jasonjmcghee 7mo ago
  > model vendors know best on giving the best harness This was only true for Claude Code for a while. Codex was poor and Gemini was unusable. Since then Codex has gotten quite good.
  - jsemrau 7mo ago
    It still fubars my code regularly at 11x the price. Github Copilot Agentic Mode + Sonnet 4.6 is stable and inexpensive.
jbergqvist 7mo ago
Would have loved to see a more detailed breakdown of performance by task type. The commit metadata is right there, seems straightforward to tag commits as feature vs refactor vs bug fix vs API change and report per-category numbers.
entrustai 7mo ago
[dead]
qsera 7mo ago
>Alibaba Group
coder_decoder 7mo ago
[flagged]
- calvinmorrison 7mo ago
  I've been.... and they genuinely... And honestly? the real x is that. it went from X to Y. dang permaban this AI slop please
- jlebensold 7mo ago
  I've been building a similar loop with jetty.io for the last few months exclusively focused on data science workflows. I think that there's a lot of hill-climbing that can be accomplished by having a clear runbook.
yuyuqueen 7mo ago
The regression rates match what I saw early on with Claude Code on my monorepo. The fix was structural, not model-level: keeping everything in a single tree (packages, tests, docs, CI config) so the agent sees downstream effects of any change. When context is split across repos, agents cheerfully break imports because they literally can't see what depends on what. Something hard to capture in benchmarks: project-level conventions. A well-maintained CLAUDE.md at the repo root — describing architecture, naming patterns, test conventions — gives the agent context it internalizes before touching code. My regression rate dropped noticeably once I started maintaining that kind of project metadata. Model choice is only half the equation — the other half is how well you've structured the information environment the agent works in.
raphaelmolly8 7mo ago
[dead]
agent5ravi 7mo ago
[flagged]
- oliver_dr 7mo ago
  [dead]
- westurner 7mo ago
  > It is understanding whether your fix preserves the invariants the original author had in mind but did not write down. This may also be the limit to the quality of an automated port to another language. What isn't encoded as automated tests or manual test procedure cannot be verified. So often I'm amazed at what it's possible to accomplish from a prompt that's certainly insufficient with insufficient context. "It should have been necessary to specify more context there," or "I would have thought that it wasn't possible to do that without reading in more context than just one source code file," and then a few prompts later, "there's where we failed for trying to skimp on context" To prevent architectural rework as a human developer also requires substantial ahead-of-time codebase review. Are AGENTS.md files the best place to summarize more comprehensive codebase review and useful dense context like guidelines for testing and architectural components in order to avoid rework?
- saltpath 7mo ago
  CI pass/fail captures regression, but there's a layer beneath it that benchmarks can't touch: what exactly did the agent submit to each external API, and can you prove it after the fact? In the benchmark context this doesn't matter everything runs locally. In production it does. The agent calls a third-party service at 2am, the service claims it returned an error, your agent retried and billed you twice. Your logs say one thing, their logs say another. The integrity problem isn't just "did the code work" it's "what was the exact request/response pair, timestamped, by whom, provably." CI solves the first. Something else has to solve the second.
smy20011 7mo ago
It interesting to see that the eval set becoming more and more expensive. Previously we just need to evaluate one test set, right now we need to create a lot of diffs and run a lot of tests.
rurban 7mo ago
The zero regression rate graph at the end is exactly my experience. Only Opus is useful right now, the rest are juniors.
devonkelley 7mo ago
[dead]
noobcoder 7mo ago
Passing tests doesn’t mean you have a working codebase. Benchmarks that rely on a fixed test suite create a real optimization problem agents (or/and even humans) learn to satisfy the tests rather than preserve the deeper properties that make the system maintainable. AI write test cases which it thinks is easier for it to satisfy and not adhere-ing to business logic We see this firsthand at Prismor with auto generated security fixes. Even with the best LLMs, validating fixes is the real bottleneck our pipeline struggles to exceed 70% on an internal golden dataset (which itself is somewhat biased). Many patches technically fix the vulnerability but introduce semantic regressions or architectural drift. Passing tests is a weak signal and proving a fix is truly safe to merge is much harder

125 points by mpweiher 8mo ago

Really long-term task benchmark showing significant improvements in very recent models, while also showing really bad regression rates across the board.

woeirua 7mo ago
Uh, Opus 4.6 avoids introducing regressions 75% of the time?
- verdverm 7mo ago
  So 1/4 times it does not introduce a regression. That's still pretty bad imo. If 1/4 commits introduced regressions, what would your team do? We are talking about regressions, what once worked no longer does, and should be measured in 9s
  - notduncansmith 7mo ago
    You overestimate many teams I think.

devcraft_ai 8mo ago

[flagged]

challengerVIE 8mo ago

To me using agents daily, the long term vision with maintainability in mind really makes the difference between us humans and agents, I like the idea. However evaluating long term maintainability over an average of just 500 loc changes does not sound like long term maintainability being measured here

KronisLV 8mo ago

> The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. This seems like a really cool thing to benchmark! Technically it'd be possible to take GitHub repos that the AI orgs probably already have, cross-reference the code against the issues and regressions, and train/validate on that. The dataset would need to be way bigger to get close to the likes of SWE-bench: https://www.swebench.com/original.html https://www.swebench.com/original.html "Vibe coded stuff gets hard to maintain and will end up buggy." Yeah, so make models that deal with that better, optimize for maintainability and consistency. Cool to see Claude doing decently though!

woadwarrior01 8mo ago
> Cool to see Claude doing decently though! The scales do seem to be tipped in its favor (cf: my other comment in this thread).

woadwarrior01 8mo ago

Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).

aurareturn 8mo ago
As far as I know, OpenAI did not release 5.3 Codex in their API. You can only use it with Codex CLI or app.
- baalimago 7mo ago
  It's there, you just need to use it with the responses API. Set model field to 'gpt-5.3-codex'
re-thc 7mo ago
5.2 and 5.2 Codex is arguably the same gen.
- jasonjmcghee 7mo ago
  Sure, but one is fine-tuned for what they are testing and one is not.

mentalgear 8mo ago

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these models with added agent tooling ("harness") ? Maybe someone has tokens to burn and can run a matrix of agent tools over the top models and provide the results?

mike_hearn 7mo ago
It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.
- pizlonator 7mo ago
  gpt-5.3 was not accessible via API, at least for me But it was in codex
  - mike_hearn 7mo ago
    Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than the underlying models.
    - pizlonator 7mo ago
      I am using responses API (these days I only use responses API when talking to OpenAI) I'll try again, maybe I was doing something else wrong, or something changed

50lo 8mo ago

It’d be interesting to see this compared against a human baseline — e.g., a competent engineer with a fixed time budget on the same tasks.

PunchyHamster 7mo ago

I'm sure with benchmarks like these future LLMs will be optimized to hide regressions by "fixing" test framework too

pixl97 7mo ago
Isn't misalignment great.

baalimago 7mo ago

Replace "Agent" with "Employee" and apply the same algorithm. Evaluate employee efficiency. Profit?

KronisLV 7mo ago
I'd unironically (and privately) want to do that with the code of both myself and those around me - to maybe see who I should listen more to, as well as who maybe less (ideally down to the feature level), because everyone has opinions, sometimes loud ones, but some approaches lead to a lot of churn and issues over the years.

gizmodo59 7mo ago

Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api https://developers.openai.com/api/docs/models/gpt-5.3-codex https://developers.openai.com/api/docs/models/gpt-5.3-codex IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if you are running any kind of agentic coding tasks. I see both Claude and gpt to be neck and neck in coding. Every other model+harness is definitely 3-6 months behind. Right now codex seems to be the best in terms of solving complex bugs, long running tasks, much higher limits and even speed while Claude seems to do well in front end and their cli ux seems nice! Codex app is very good though (wish it wasn’t electron as a memory hog but it’s good)

p1esk 7mo ago
Are you saying they did not use native harnesses like Claude Code or Codex? How did they do it then?
jasonjmcghee 7mo ago
> model vendors know best on giving the best harness This was only true for Claude Code for a while. Codex was poor and Gemini was unusable. Since then Codex has gotten quite good.
- jsemrau 7mo ago
  It still fubars my code regularly at 11x the price. Github Copilot Agentic Mode + Sonnet 4.6 is stable and inexpensive.

jbergqvist 7mo ago

Would have loved to see a more detailed breakdown of performance by task type. The commit metadata is right there, seems straightforward to tag commits as feature vs refactor vs bug fix vs API change and report per-category numbers.

entrustai 7mo ago

[dead]

qsera 7mo ago

>Alibaba Group

coder_decoder 7mo ago

[flagged]

calvinmorrison 7mo ago
I've been.... and they genuinely... And honestly? the real x is that. it went from X to Y. dang permaban this AI slop please
jlebensold 7mo ago
I've been building a similar loop with jetty.io for the last few months exclusively focused on data science workflows. I think that there's a lot of hill-climbing that can be accomplished by having a clear runbook.

yuyuqueen 7mo ago

The regression rates match what I saw early on with Claude Code on my monorepo. The fix was structural, not model-level: keeping everything in a single tree (packages, tests, docs, CI config) so the agent sees downstream effects of any change. When context is split across repos, agents cheerfully break imports because they literally can't see what depends on what. Something hard to capture in benchmarks: project-level conventions. A well-maintained CLAUDE.md at the repo root — describing architecture, naming patterns, test conventions — gives the agent context it internalizes before touching code. My regression rate dropped noticeably once I started maintaining that kind of project metadata. Model choice is only half the equation — the other half is how well you've structured the information environment the agent works in.

raphaelmolly8 7mo ago

[dead]

agent5ravi 7mo ago

[flagged]

oliver_dr 7mo ago
[dead]
westurner 7mo ago
> It is understanding whether your fix preserves the invariants the original author had in mind but did not write down. This may also be the limit to the quality of an automated port to another language. What isn't encoded as automated tests or manual test procedure cannot be verified. So often I'm amazed at what it's possible to accomplish from a prompt that's certainly insufficient with insufficient context. "It should have been necessary to specify more context there," or "I would have thought that it wasn't possible to do that without reading in more context than just one source code file," and then a few prompts later, "there's where we failed for trying to skimp on context" To prevent architectural rework as a human developer also requires substantial ahead-of-time codebase review. Are AGENTS.md files the best place to summarize more comprehensive codebase review and useful dense context like guidelines for testing and architectural components in order to avoid rework?
saltpath 7mo ago
CI pass/fail captures regression, but there's a layer beneath it that benchmarks can't touch: what exactly did the agent submit to each external API, and can you prove it after the fact? In the benchmark context this doesn't matter everything runs locally. In production it does. The agent calls a third-party service at 2am, the service claims it returned an error, your agent retried and billed you twice. Your logs say one thing, their logs say another. The integrity problem isn't just "did the code work" it's "what was the exact request/response pair, timestamped, by whom, provably." CI solves the first. Something else has to solve the second.

smy20011 7mo ago

It interesting to see that the eval set becoming more and more expensive. Previously we just need to evaluate one test set, right now we need to create a lot of diffs and run a lot of tests.

rurban 7mo ago

The zero regression rate graph at the end is exactly my experience. Only Opus is useful right now, the rest are juniors.

devonkelley 7mo ago

[dead]

noobcoder 7mo ago

Passing tests doesn’t mean you have a working codebase. Benchmarks that rely on a fixed test suite create a real optimization problem agents (or/and even humans) learn to satisfy the tests rather than preserve the deeper properties that make the system maintainable. AI write test cases which it thinks is easier for it to satisfy and not adhere-ing to business logic We see this firsthand at Prismor with auto generated security fixes. Even with the best LLMs, validating fixes is the real bottleneck our pipeline struggles to exceed 70% on an internal golden dataset (which itself is somewhat biased). Many patches technically fix the vulnerability but introduce semantic regressions or architectural drift. Passing tests is a weak signal and proving a fix is truly safe to merge is much harder