15 ms·

>Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. A hidden metric here is the number of new bugs created by these fixes.

by qsera 2mo ago

>Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. A hidden metric here is the number of new bugs created by these fixes. If an LLM creates 10x bugs but create 5x more new bugs, then it is not really an improvement. Because that is 5x more bugs that the user has to observe and report (unless it is a security vulnerability, which the LLMs might detect) before it can be fixed.

ibobev 2mo ago
> A hidden metric here is the number of new bugs created by these fixes. If the project has proper test coverage, there should be no significant number of new bugs. This is no different than the possible regressions added by a human-implemented feature. And LLMs have made implementing a massive number of tests far easier and faster than before the LLM era.
socalgal2 2mo ago
This is effectively a claim that Mozillia, Apple, Google's developers are idiots and irresponsible and just accepted the fixes without checking them. That's a pretty strong claim which I think you'd have a hard time proving or even finding any evidence for
- qsera 2mo ago
  I mean no such claim. One does not have to be an idiot or irresponsible to overlook an unintended consequence of some code change in a very large, legacy code base.
  - j_w 2mo ago
    If that claim were even reasonable in the first place you would have to accept that they were idiots long before by allowing bugs to be introduced in the first place.
    - stochasto 2mo ago
      The presence or introduction of bugs in software does not mean it’s developers are ‘idiots’ unless practically everyone who has ever written code is an ‘idiot’
      - j_w 2mo ago
        Yes I didn't say this. I'm using it to show how absurd the original claim was.
- archagon 2mo ago
  I think LLMs are particularly effective at turning once-competent developers into idiots. (“Just ask Claude.”)
skydhash 2mo ago
Also there's the "let's make time to work on those bugs by using this new tool". Time that wouldn't have been allocated otherwise.
- capr 2mo ago
  ...allocated on not fixing the bugs, sure. and AI does not decide what you spend your time on.
- socalgal2 2mo ago
  The LLMs find the bugs and the users will get exploited unless fixed. Sure am glad the choose fix them.

by qsera 2mo ago

ibobev 2mo ago

> A hidden metric here is the number of new bugs created by these fixes. If the project has proper test coverage, there should be no significant number of new bugs. This is no different than the possible regressions added by a human-implemented feature. And LLMs have made implementing a massive number of tests far easier and faster than before the LLM era.

socalgal2 2mo ago

This is effectively a claim that Mozillia, Apple, Google's developers are idiots and irresponsible and just accepted the fixes without checking them. That's a pretty strong claim which I think you'd have a hard time proving or even finding any evidence for

qsera 2mo ago
I mean no such claim. One does not have to be an idiot or irresponsible to overlook an unintended consequence of some code change in a very large, legacy code base.
- j_w 2mo ago
  If that claim were even reasonable in the first place you would have to accept that they were idiots long before by allowing bugs to be introduced in the first place.
  - stochasto 2mo ago
    The presence or introduction of bugs in software does not mean it’s developers are ‘idiots’ unless practically everyone who has ever written code is an ‘idiot’
    - j_w 2mo ago
      Yes I didn't say this. I'm using it to show how absurd the original claim was.
archagon 2mo ago
I think LLMs are particularly effective at turning once-competent developers into idiots. (“Just ask Claude.”)

skydhash 2mo ago

Also there's the "let's make time to work on those bugs by using this new tool". Time that wouldn't have been allocated otherwise.

capr 2mo ago
...allocated on not fixing the bugs, sure. and AI does not decide what you spend your time on.
socalgal2 2mo ago
The LLMs find the bugs and the users will get exploited unless fixed. Sure am glad the choose fix them.