3 ms·

I would be interested to see how it performs on end-user software where bug reports are nebulous at best, ridiculous at worst. Furthermore, most of those fixes

I would be interested to see how it performs on end-user software where bug reports are nebulous at best, ridiculous at worst. Furthermore, most of those fixes tend to be upstream bugs and very rarely anything to do with the actual software.

abhikrc 3y ago
The entire setup is available for inspection, please see https://github.com/nus-apr/auto-code-rover https://github.com/nus-apr/auto-code-rover if you need example bugs we can provide that too. Some examples also appear in the arxiv paper, please see https://arxiv.org/abs/2404.05427 https://arxiv.org/abs/2404.05427
- joenot443 3y ago
  This is super fascinating stuff, excellent work. As most of us don't have the time to read the entirety of the paper, are you able to directly link to some issues which have been landed and closed? Some personal favorites would be awesome :) I think I speak for others when I say the best way to judge the efficacy of this project is some real-world, on-site examples of it being used in prod. I'm especially curious for its performance in feature-request or flakey bug report type issues as opposed to reliable test failures. I expect the former is much tougher!
  - dboreham 3y ago
    fwiw the example issue highlighted in the post was already fixed by a human 3 years ago so I wouldn't expect to see much in the way of real life fixed issues yet.
  - yuntong 3y ago
    Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover: - From sympy: https://github.com/sympy/sympy/issues/13643 https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-2/applicable_patch/sympy__sympy-13647_2024-04-04_13-43-51/extracted_patch_1.diff https://github.com/nus-apr/auto-code-rover/blob/main/results... - Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070 https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-2/applicable_patch/scikit-learn__scikit-learn-13142_2024-04-04_13-28-08/extracted_patch_1.diff https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment. There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results https://github.com/nus-apr/auto-code-rover/tree/main/results).

abhikrc 3y ago

The entire setup is available for inspection, please see https://github.com/nus-apr/auto-code-rover https://github.com/nus-apr/auto-code-rover if you need example bugs we can provide that too. Some examples also appear in the arxiv paper, please see https://arxiv.org/abs/2404.05427 https://arxiv.org/abs/2404.05427

joenot443 3y ago
This is super fascinating stuff, excellent work. As most of us don't have the time to read the entirety of the paper, are you able to directly link to some issues which have been landed and closed? Some personal favorites would be awesome :) I think I speak for others when I say the best way to judge the efficacy of this project is some real-world, on-site examples of it being used in prod. I'm especially curious for its performance in feature-request or flakey bug report type issues as opposed to reliable test failures. I expect the former is much tougher!
- dboreham 3y ago
  fwiw the example issue highlighted in the post was already fixed by a human 3 years ago so I wouldn't expect to see much in the way of real life fixed issues yet.
- yuntong 3y ago
  Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover: - From sympy: https://github.com/sympy/sympy/issues/13643 https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-2/applicable_patch/sympy__sympy-13647_2024-04-04_13-43-51/extracted_patch_1.diff https://github.com/nus-apr/auto-code-rover/blob/main/results... - Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070 https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results/acr-run-2/applicable_patch/scikit-learn__scikit-learn-13142_2024-04-04_13-28-08/extracted_patch_1.diff https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment. There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results https://github.com/nus-apr/auto-code-rover/tree/main/results).