6 ms·

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild a

by dimal 3y ago

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug is in figuring out what’s causing the bug. If you know it’s caused by an incorrect operation, and we know that LLMs can fix simple defects like this, what does this prove? Has anyone dug through the paper yet to see what the rest of the issues look like? And what the diffs look like? I suppose I’ll try when I have a sec.

drcode 3y ago
> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.
- dimal 3y ago
  I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you’d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn’t need automation.
  - hvis 3y ago
    To be fair, sometimes meticulous users investigate the bugs and write down logical chains explaining the causes and even offer a solution at the end (which they can't apply for the lack of commit access, for instance). The proposed solution isn't always right, of course, but it would be incorrect to say that no bug reports come with a diagnosed cause. But that's exactly where a conscious reviewer is most needed, I believe.
    - citrin_ru 3y ago
      I sometimes write a detailed bug reports but not a PR when there are different ways to address the problem (and all look bad to me) or the fix can introduce new problems. But I would expect LLM to ignore tradeoffs and choose an option which not necessarily the best for the same reason I hesitate - luck of understanding of this specific project.
- skywhopper 3y ago
  It fixes 12% of their benchmark suite, not 12% of bug reports.
- medellin 3y ago
  In my 15 years i would say less than 1% of bug reports are like this. If you know the bug to this level most people just would fix it themselves
- aiauthoritydev2 3y ago
  12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.
  - littlestymaar 3y ago
    Except this is automated, so you could get multiples orders of magnitude more bug filled, so you need to have a very low false positive ratio to avoid being overwhelmed by automatically generated crap (which is basically spam).
  - sitkack 3y ago
    Have the LLM rewrite the bug reports.
    - killingtime74 3y ago
      Why not have LLM write AGI while you're at it
      - sitkack 3y ago
        It is and it will!
    - fasa99 3y ago
      You'd want three LLMs, one to create the bugs, one to report it, one to fix it. I joke of course but on the other hand this is potentially a worthwhile architecture from a self-training perspective - a bug-creating LLM means your training set size is as big as you want it +/- GAN features.
megablast 3y ago
Exactly. This is not perfect and doesn't fix every report so it is useless.
- skywhopper 3y ago
  On the contrary, it’s worse than useless. If it could fix 12% of bugs (it can’t — it only fixes 12% of their benchmark suite), you’d still have to figure out which 12% of the responses it gave were good. So, 88% of the time you’d have wasted time confirming a “fix” that doesn’t work. But it’s worse than that. Because even on the fixes it got right, you’d still have to fully vet it, because this tool doesn’t know when it can’t solve something, or ask for clarification. It just gives a wrong answer. So you didn’t save 12% of your effort, you wasted probably more than double your effort checking the work of a tool that is wrong eight out of nine times.
- dimal 3y ago
  That’s not what I said and you know it. I’m not saying LLMs are useless. I’m not even saying this tool is useless. I’m saying I’m not impressed with this tool, at least as represented in the demo.
codeonline 3y ago
I agree that bugs aren't as well specified as the example. But a specification for a new feature certainly can be. I'm going to give it a try on my side project and see if it can at least provide a hint or some guidance on the development of small new features in an existing well structured project.
bee_rider 3y ago
Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.
- bfdm 3y ago
  While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.
  - throwup238 3y ago
    I think that depends on the exact KPI.
    - ffsm8 3y ago
      KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work. The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship
      - shabble 3y ago
        "ratio of open/total issues" can definitely be gamed by autoclosing anything that isn't an easy fix. "average time to resolution" is also susceptible. Both of these are pretty common all over the place, including OSS e.g. https://isitmaintained.com/#metrics https://isitmaintained.com/#metrics I suspect this sort of thing is one of the major motivations for the (as a user/reporter) infuriating rise in automated "this bug hasn't been touched in NN days, autoclosing for staleness" bots on various issue trackers.
        bee_rider 3y ago
stingraycharles 3y ago
It appears that they’re using the PRs from the top5000 most popular PyPi packages for their bench: https://github.com/princeton-nlp/SWE-bench/tree/main/swebench/collect https://github.com/princeton-nlp/SWE-bench/tree/main/swebenc...
forty 3y ago
The trick is that people would use LLM to write very long and detailed bug reports :p
jcarrano 3y ago
Maybe it would be better if the agent would help people submit better reports instead of trying to fix it. E.g. it could ask them to add missing information, test different combinations of inputs, etc. I could also learn which maintainer to ping according to the type of issue.
gorjusborg 3y ago
If the bug report needs to be of a certain quality to work, they've just invented issue-oriented programming.
chinchilla2020 3y ago
Agreed. I have never encountered a simple math bug in the wild. To a non-programmer, putting in tests for myfunc(x) {return x + 2;} sounds useful but in reality computers do not tend to have any issues performing basic algebra.

by dimal 3y ago

drcode 3y ago

> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

dimal 3y ago
I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you’d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn’t need automation.
- hvis 3y ago
  To be fair, sometimes meticulous users investigate the bugs and write down logical chains explaining the causes and even offer a solution at the end (which they can't apply for the lack of commit access, for instance). The proposed solution isn't always right, of course, but it would be incorrect to say that no bug reports come with a diagnosed cause. But that's exactly where a conscious reviewer is most needed, I believe.
  - citrin_ru 3y ago
    I sometimes write a detailed bug reports but not a PR when there are different ways to address the problem (and all look bad to me) or the fix can introduce new problems. But I would expect LLM to ignore tradeoffs and choose an option which not necessarily the best for the same reason I hesitate - luck of understanding of this specific project.
skywhopper 3y ago
It fixes 12% of their benchmark suite, not 12% of bug reports.
medellin 3y ago
In my 15 years i would say less than 1% of bug reports are like this. If you know the bug to this level most people just would fix it themselves
aiauthoritydev2 3y ago
12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.
- littlestymaar 3y ago
  Except this is automated, so you could get multiples orders of magnitude more bug filled, so you need to have a very low false positive ratio to avoid being overwhelmed by automatically generated crap (which is basically spam).
- sitkack 3y ago
  Have the LLM rewrite the bug reports.
  - killingtime74 3y ago
    Why not have LLM write AGI while you're at it
    - sitkack 3y ago
      It is and it will!
  - fasa99 3y ago
    You'd want three LLMs, one to create the bugs, one to report it, one to fix it. I joke of course but on the other hand this is potentially a worthwhile architecture from a self-training perspective - a bug-creating LLM means your training set size is as big as you want it +/- GAN features.

megablast 3y ago

Exactly. This is not perfect and doesn't fix every report so it is useless.

skywhopper 3y ago
On the contrary, it’s worse than useless. If it could fix 12% of bugs (it can’t — it only fixes 12% of their benchmark suite), you’d still have to figure out which 12% of the responses it gave were good. So, 88% of the time you’d have wasted time confirming a “fix” that doesn’t work. But it’s worse than that. Because even on the fixes it got right, you’d still have to fully vet it, because this tool doesn’t know when it can’t solve something, or ask for clarification. It just gives a wrong answer. So you didn’t save 12% of your effort, you wasted probably more than double your effort checking the work of a tool that is wrong eight out of nine times.
dimal 3y ago
That’s not what I said and you know it. I’m not saying LLMs are useless. I’m not even saying this tool is useless. I’m saying I’m not impressed with this tool, at least as represented in the demo.

codeonline 3y ago

I agree that bugs aren't as well specified as the example. But a specification for a new feature certainly can be. I'm going to give it a try on my side project and see if it can at least provide a hint or some guidance on the development of small new features in an existing well structured project.

bee_rider 3y ago

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

bfdm 3y ago
While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.
- throwup238 3y ago
  I think that depends on the exact KPI.
  - ffsm8 3y ago
    KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work. The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship
    - shabble 3y ago
      "ratio of open/total issues" can definitely be gamed by autoclosing anything that isn't an easy fix. "average time to resolution" is also susceptible. Both of these are pretty common all over the place, including OSS e.g. https://isitmaintained.com/#metrics https://isitmaintained.com/#metrics I suspect this sort of thing is one of the major motivations for the (as a user/reporter) infuriating rise in automated "this bug hasn't been touched in NN days, autoclosing for staleness" bots on various issue trackers.
      - bee_rider 3y ago

stingraycharles 3y ago

It appears that they’re using the PRs from the top5000 most popular PyPi packages for their bench: https://github.com/princeton-nlp/SWE-bench/tree/main/swebench/collect https://github.com/princeton-nlp/SWE-bench/tree/main/swebenc...

forty 3y ago

The trick is that people would use LLM to write very long and detailed bug reports :p

jcarrano 3y ago

Maybe it would be better if the agent would help people submit better reports instead of trying to fix it. E.g. it could ask them to add missing information, test different combinations of inputs, etc. I could also learn which maintainer to ping according to the type of issue.

gorjusborg 3y ago

If the bug report needs to be of a certain quality to work, they've just invented issue-oriented programming.

chinchilla2020 3y ago

Agreed. I have never encountered a simple math bug in the wild. To a non-programmer, putting in tests for myfunc(x) {return x + 2;} sounds useful but in reality computers do not tend to have any issues performing basic algebra.