4 ms·

And the other 78% of time it just creates a bunch of noise that someone has to sift through?

egeozcan 3y ago
That's in my experience better than the percentage of although usually good-intentioned but nevertheless unusable PRs popular repositories get.
ectopasm83 3y ago
The point is that the success rate is progressing, paper after paper > The baseline results of Magis (10%), Devin (14%) are evaluated in another subset of SWE-bench, which we cannot directly compare with, so we take the results from their technical reports as a reference. Wondering how it compares with these models.
- invalidusernam3 3y ago
  Why not use AutoCodeRover, Magis, and Devin together for 46% /s
arp242 3y ago
Here's a list of all the successful and unsuccessful patches: https://gist.github.com/arp242/0dc5dab0f7cd10e663cfc26866651528 https://gist.github.com/arp242/0dc5dab0f7cd10e663cfc26866651... Ideally, it should also include the problem statement, but that's not in their JSON file and can't arsed to continue working on it – it's just a quick script I cooked up. I find it very hard to judge the quality of most of these patches because I'm not familiar with these projects. However, looking at the SWE-bench dataset I don't think it's representative of real-world issues, so "22% of real-world GitHub issues" is not really accurate regardless.
- wsdookadr 3y ago
  What makes you say it's not representative?
  - arp242 3y ago
    Look at the data. Does that seem like the average bug report to you?
    - falcor84 3y ago
      It would help if you were to provide a specific example or two
      - arp242 3y ago
        You can't demonstrate whether a dataset is representative or not by "an example or two". You need to look at all the data. And all of this is fine. It's just a benchmark suit and doesn't need to be fully representative. The dataset itself doesn't even claim to be that as far as I can find. All I'm saying that the title wasn't really accurate.
  - skywhopper 3y ago
    SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects. Here are some rules they used to trim down the SWE-bench Lite problems: * We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues. * We remove instances that have fewer than 40 words in the problem statement. * We remove instances that edit more than 1 file. * We remove instances where the gold patch has more than 3 edit hunks (see patch). See https://www.swebench.com/lite.html https://www.swebench.com/lite.html
    - kevindamm 3y ago
      That's... rather limiting.
- yuntong 3y ago
  The problem statement of each issue is included in each result folder as `problem_statement.txt` (such as: https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-1/applicable_patch/astropy__astropy-12907_2024-04-06_13-51-50/problem_statement.txt https://github.com/nus-apr/auto-code-rover/blob/main/results...). The developer patch for each issue is similarly included as `developer_patch.diff`.
yourapostasy 3y ago
In short, no. The ArXiv paper mentions the human developer must supply a unit test (which can conceivably be coded with at least the assistance of an AI agent if not autonomously coded, but their experiment relies upon the former kind of unit test) that issues a pass-fail signal. So the 78% of failures are clearly identified, at the cost of implementing TDD for the Issue. The side effects story is punted upon, but I’d still take this over the nothing we have today. Of course, over a relatively short amount of time using this, I’d expect to experience the 22% (or whatever the real rate is) success rate to drop asymptotically towards zero as the low hanging fruit of the approach are mined out and it becomes kind of like another linter in our CICD pipelines. The impact of this tooling upon staff skills development will be interesting to say the least.
- yuntong 3y ago
  AutoCodeRover does not require or assume a unit test to generate patches. The results discussed in Section 6.1 of the ArXiv paper are generated without any unit test. The unit tests are used by SWE-bench, when evaluating the correctness of AutoCodeRover-generated patches. That being said, when some unit tests are available (either written by developers or with assistance from other tools), AutoCodeRover can make use of them to perform some analysis like Spectrum-based Fault Localization (SBFL). This kind of analysis output can help the agent in pinpointing locations to fix. (Please see Section 6.2 for the analysis on SBFL.)
  - dboreham 3y ago
    > AutoCodeRover does not require or assume a unit test to generate patches. You have this backwards : it's traditional (at least in the past 15 years or so) to have a test to go along with every code change. The idea is that the test proves a) the bug existed prior to the fix and b) the bug is not there after the fix is applied. Commenters here are noting that ACR generates fixes but not tests.
    - yuntong 3y ago
      The previous comment was to describe the experiment settings. AutoCodeRover currently generates patches. Auto-generating high quality tests can be a parallel effort and another direction to explore. These efforts can eventually be used together.
wsdookadr 3y ago
There's actual SWE jobs where humans sift through this kind of noise. Someone told me they worked such a job recently. It's a good tool to add pressure and raise expectations. Maybe this is the future..
- skywhopper 3y ago
  They only know the 22% number because unit tests to check for a fix are included in the benchmark. In other words, in a real world situation, the human would still need to double check. The patches this tool generates do not include appropriate tests or explanations and would never pass code review by a qualified human.
DalasNoin 3y ago
Just about a week ago open devin got about 13% on this benchmark. Just give it a few more weeks. edit: apparently it's not the exact same benchmark but a similar one
- moolcool 3y ago
  If it continues at this pace, then it'll solve 108% of GitHub issues in just 3 months
Morelesshell 3y ago
if a ticket is open and AutoCodeRover just says "was unable to find something" its still better to have 22% fixed automatically.
- skywhopper 3y ago
  But it doesn’t say that. It submits a patch that doesn’t solve the problem instead.
- maleldil 3y ago
  LLMs are unable to say that they don't know something. They just generate nonsense.
deleted 3y ago
[deleted]
skywhopper 3y ago
Yes, and to be clear, the benchmark used here is merely the 300 simplest problems in the larger benchmark suite, which itself is only a tiny subset of issues from a dozen large (and presumably well-curated) Python projects. Not to mention that making the code fix is only a tiny part of resolving an issue. There should also be explanations and added test cases. In other words, I doubt the 22% of “fixes” would pass review by the project owner if a human submitted them.
deleted 3y ago
[deleted]

That's in my experience better than the percentage of although usually good-intentioned but nevertheless unusable PRs popular repositories get.

ectopasm83 3y ago

The point is that the success rate is progressing, paper after paper > The baseline results of Magis (10%), Devin (14%) are evaluated in another subset of SWE-bench, which we cannot directly compare with, so we take the results from their technical reports as a reference. Wondering how it compares with these models.

invalidusernam3 3y ago
Why not use AutoCodeRover, Magis, and Devin together for 46% /s

arp242 3y ago

Here's a list of all the successful and unsuccessful patches: https://gist.github.com/arp242/0dc5dab0f7cd10e663cfc26866651528 https://gist.github.com/arp242/0dc5dab0f7cd10e663cfc26866651... Ideally, it should also include the problem statement, but that's not in their JSON file and can't arsed to continue working on it – it's just a quick script I cooked up. I find it very hard to judge the quality of most of these patches because I'm not familiar with these projects. However, looking at the SWE-bench dataset I don't think it's representative of real-world issues, so "22% of real-world GitHub issues" is not really accurate regardless.

wsdookadr 3y ago
What makes you say it's not representative?
- arp242 3y ago
  Look at the data. Does that seem like the average bug report to you?
  - falcor84 3y ago
    It would help if you were to provide a specific example or two
    - arp242 3y ago
      You can't demonstrate whether a dataset is representative or not by "an example or two". You need to look at all the data. And all of this is fine. It's just a benchmark suit and doesn't need to be fully representative. The dataset itself doesn't even claim to be that as far as I can find. All I'm saying that the title wasn't really accurate.
- skywhopper 3y ago
  SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects. Here are some rules they used to trim down the SWE-bench Lite problems: * We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues. * We remove instances that have fewer than 40 words in the problem statement. * We remove instances that edit more than 1 file. * We remove instances where the gold patch has more than 3 edit hunks (see patch). See https://www.swebench.com/lite.html https://www.swebench.com/lite.html
  - kevindamm 3y ago
    That's... rather limiting.
yuntong 3y ago
The problem statement of each issue is included in each result folder as `problem_statement.txt` (such as: https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-1/applicable_patch/astropy__astropy-12907_2024-04-06_13-51-50/problem_statement.txt https://github.com/nus-apr/auto-code-rover/blob/main/results...). The developer patch for each issue is similarly included as `developer_patch.diff`.

yourapostasy 3y ago

In short, no. The ArXiv paper mentions the human developer must supply a unit test (which can conceivably be coded with at least the assistance of an AI agent if not autonomously coded, but their experiment relies upon the former kind of unit test) that issues a pass-fail signal. So the 78% of failures are clearly identified, at the cost of implementing TDD for the Issue. The side effects story is punted upon, but I’d still take this over the nothing we have today. Of course, over a relatively short amount of time using this, I’d expect to experience the 22% (or whatever the real rate is) success rate to drop asymptotically towards zero as the low hanging fruit of the approach are mined out and it becomes kind of like another linter in our CICD pipelines. The impact of this tooling upon staff skills development will be interesting to say the least.

yuntong 3y ago
AutoCodeRover does not require or assume a unit test to generate patches. The results discussed in Section 6.1 of the ArXiv paper are generated without any unit test. The unit tests are used by SWE-bench, when evaluating the correctness of AutoCodeRover-generated patches. That being said, when some unit tests are available (either written by developers or with assistance from other tools), AutoCodeRover can make use of them to perform some analysis like Spectrum-based Fault Localization (SBFL). This kind of analysis output can help the agent in pinpointing locations to fix. (Please see Section 6.2 for the analysis on SBFL.)
- dboreham 3y ago
  > AutoCodeRover does not require or assume a unit test to generate patches. You have this backwards : it's traditional (at least in the past 15 years or so) to have a test to go along with every code change. The idea is that the test proves a) the bug existed prior to the fix and b) the bug is not there after the fix is applied. Commenters here are noting that ACR generates fixes but not tests.
  - yuntong 3y ago
    The previous comment was to describe the experiment settings. AutoCodeRover currently generates patches. Auto-generating high quality tests can be a parallel effort and another direction to explore. These efforts can eventually be used together.

wsdookadr 3y ago

There's actual SWE jobs where humans sift through this kind of noise. Someone told me they worked such a job recently. It's a good tool to add pressure and raise expectations. Maybe this is the future..

skywhopper 3y ago
They only know the 22% number because unit tests to check for a fix are included in the benchmark. In other words, in a real world situation, the human would still need to double check. The patches this tool generates do not include appropriate tests or explanations and would never pass code review by a qualified human.

DalasNoin 3y ago

Just about a week ago open devin got about 13% on this benchmark. Just give it a few more weeks. edit: apparently it's not the exact same benchmark but a similar one

moolcool 3y ago
If it continues at this pace, then it'll solve 108% of GitHub issues in just 3 months

Morelesshell 3y ago

if a ticket is open and AutoCodeRover just says "was unable to find something" its still better to have 22% fixed automatically.

skywhopper 3y ago
But it doesn’t say that. It submits a patch that doesn’t solve the problem instead.
maleldil 3y ago
LLMs are unable to say that they don't know something. They just generate nonsense.

deleted 3y ago

[deleted]

skywhopper 3y ago

Yes, and to be clear, the benchmark used here is merely the 300 simplest problems in the larger benchmark suite, which itself is only a tiny subset of issues from a dozen large (and presumably well-curated) Python projects. Not to mention that making the code fix is only a tiny part of resolving an issue. There should also be explanations and added test cases. In other words, I doubt the 22% of “fixes” would pass review by the project owner if a human submitted them.

deleted 3y ago

[deleted]