5 ms·

> We are 4 years into this ‘revolution’ and the industry has almost nothing to show for it. Software has not improved in quality, Apple, Mozillia, and Firefox

> We are 4 years into this ‘revolution’ and the industry has almost nothing to show for it. Software has not improved in quality, Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. So this assertion that Software as not improved in quality is arguably if not provably false. There are 1000s of other examples. I have my own. Personal projects that were stopped because of dependency rot, having 100s of bugs fixed, nearly all dependencies removed, and lots of tests added just by asking. I know tons of others with similar experiences > I do not feel like I’m falling behind by not using them Obviously you are correct about "how you feel". But, this argument to me is more like a horse owner claming they don't feel like they're failling behind to trucks that can carry 10x, 100x, 1000x more than their horse. Sure, maybe the horse can go places the truck can't. For most uses the truck is better than the horse. > The LLM-generated PRs I get are still rubbish. I can't argue with that. I haven't seed the LLM-generated PRs "you" get. I can say though the LLM-generated bug reports I've seen are 1000x better than human ones. Human bug report "your software breaks, fix it!". LLM bug report "There is a bug in this specfic part of the code for these specific reasons and here's a repo I wrote to demonstrate the bug. Here's instructions on how to run the repo. There's the code to fix the bug. And, here's a test to show the bug is fixed." Is it perfect? No, sometimes the bug isn't real because the code can only be reached in situtations that can never happen. But still, 7 of 8 times it's a real bug and the bug report are the best I've ever received. you can go through this list: https://www.firefox.com/en-US/firefox/150.0/releasenotes/ https://www.firefox.com/en-US/firefox/150.0/releasenotes/ and for those very few issues not still blocked you can read the examples

gr_norm 2mo ago
LLMs are very helpful as a debugging aid, yes, but in large part because the fixes tend to be small and verifiable. That this does not carry over to many other use cases is the crux of the problem. I myself use them to accelerate programming tasks, so I'm not anywhere near as pessimistic as the author, but the claimed multiples of productivity definitely haven't materialized for me.
qsera 2mo ago
>Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. A hidden metric here is the number of new bugs created by these fixes. If an LLM creates 10x bugs but create 5x more new bugs, then it is not really an improvement. Because that is 5x more bugs that the user has to observe and report (unless it is a security vulnerability, which the LLMs might detect) before it can be fixed.
- ibobev 2mo ago
  > A hidden metric here is the number of new bugs created by these fixes. If the project has proper test coverage, there should be no significant number of new bugs. This is no different than the possible regressions added by a human-implemented feature. And LLMs have made implementing a massive number of tests far easier and faster than before the LLM era.
- socalgal2 2mo ago
  This is effectively a claim that Mozillia, Apple, Google's developers are idiots and irresponsible and just accepted the fixes without checking them. That's a pretty strong claim which I think you'd have a hard time proving or even finding any evidence for
  - qsera 2mo ago
    I mean no such claim. One does not have to be an idiot or irresponsible to overlook an unintended consequence of some code change in a very large, legacy code base.
    - j_w 2mo ago
      If that claim were even reasonable in the first place you would have to accept that they were idiots long before by allowing bugs to be introduced in the first place.
      - stochasto 2mo ago
        The presence or introduction of bugs in software does not mean it’s developers are ‘idiots’ unless practically everyone who has ever written code is an ‘idiot’
bendmorris 2mo ago
>For most uses the truck is better than the horse. I think you're talking past the author here, who says: >By not leaning into LLMs I’ve given myself the room to specialise in a smaller set of skills. So far that conscious decision is working out well, and I remain confident about my job security. "Most uses" might mean boilerplate and simple frontend work that LLMs can do easily because it's formulaic and doesn't require much creativity. OP is simply not doing that type of work. I don't think anyone sane is really claiming 100x or 1000x speedups anymore. Most people I talk to who use LLMs are closer to 1.5 or 2. Because their job is not mostly boilerplate, there is simply not that much room to be improved.
- deleted 2mo ago
  [deleted]
- bensyverson 2mo ago
  It’s not about speed-ups, it’s the fact that I’ll attempt a project far more ambitious than I would have in the past. Over the past two months I’ve built out a very complex web app for a client. Without the aid of AI it would be far smaller, simpler and less capable. And honestly I probably wouldn’t have agreed to take on the job in the first place.
  - lproven 2mo ago
    > far smaller, simpler and less capable So, better, in other words?
    - bensyverson 2mo ago
      Sure, in the same way that nano is better than Microsoft Word. Unfortunately, sometimes nano is not enough. In the case of this client project, we are only now getting to enough functionality to be generally useful. The complexity that exists in the project is necessary complexity. A lot of coders are simply wrong about complexity, and feel it can always be reduced.
      - 2mo ago
iammrpayments 2mo ago
Not the 1000x more productive numbers again please. Is this a meme or something? If you said +17% I might have taken you seriously.
- socalgal2 2mo ago
  I gave the 1000x example because a single semi-truck can generally carry 240x what a horse can carry. A horse can carry 150-200lbs. A semi-truck can carry 42,000 to 48,000 pounds. The largest truck carries 2.2 million pounds or 11000x the horse. To see an example of this 1000x, all we have to do is look at the flood of bug reports and fixes to browsers and OSes. Does it do 1000x for every project? No, and neither does a pickup truck do 1000x what a horse does. But, even a small pickup truck does more than a horse for most use cases (carrying people or stuff) and similarly, LLMs do more for most use cases.
lproven 2mo ago
> Obviously you are correct about "how you feel". I posted the link. I didn't write it. My name is Liam Proven, thus "lproven", while they are Joshua Barretto, thus "jsbarretto".

gr_norm 2mo ago

LLMs are very helpful as a debugging aid, yes, but in large part because the fixes tend to be small and verifiable. That this does not carry over to many other use cases is the crux of the problem. I myself use them to accelerate programming tasks, so I'm not anywhere near as pessimistic as the author, but the claimed multiples of productivity definitely haven't materialized for me.

qsera 2mo ago

>Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. A hidden metric here is the number of new bugs created by these fixes. If an LLM creates 10x bugs but create 5x more new bugs, then it is not really an improvement. Because that is 5x more bugs that the user has to observe and report (unless it is a security vulnerability, which the LLMs might detect) before it can be fixed.

ibobev 2mo ago
> A hidden metric here is the number of new bugs created by these fixes. If the project has proper test coverage, there should be no significant number of new bugs. This is no different than the possible regressions added by a human-implemented feature. And LLMs have made implementing a massive number of tests far easier and faster than before the LLM era.
socalgal2 2mo ago
This is effectively a claim that Mozillia, Apple, Google's developers are idiots and irresponsible and just accepted the fixes without checking them. That's a pretty strong claim which I think you'd have a hard time proving or even finding any evidence for
- qsera 2mo ago
  I mean no such claim. One does not have to be an idiot or irresponsible to overlook an unintended consequence of some code change in a very large, legacy code base.
  - j_w 2mo ago
    If that claim were even reasonable in the first place you would have to accept that they were idiots long before by allowing bugs to be introduced in the first place.
    - stochasto 2mo ago
      The presence or introduction of bugs in software does not mean it’s developers are ‘idiots’ unless practically everyone who has ever written code is an ‘idiot’

bendmorris 2mo ago

>For most uses the truck is better than the horse. I think you're talking past the author here, who says: >By not leaning into LLMs I’ve given myself the room to specialise in a smaller set of skills. So far that conscious decision is working out well, and I remain confident about my job security. "Most uses" might mean boilerplate and simple frontend work that LLMs can do easily because it's formulaic and doesn't require much creativity. OP is simply not doing that type of work. I don't think anyone sane is really claiming 100x or 1000x speedups anymore. Most people I talk to who use LLMs are closer to 1.5 or 2. Because their job is not mostly boilerplate, there is simply not that much room to be improved.

deleted 2mo ago
[deleted]
bensyverson 2mo ago
It’s not about speed-ups, it’s the fact that I’ll attempt a project far more ambitious than I would have in the past. Over the past two months I’ve built out a very complex web app for a client. Without the aid of AI it would be far smaller, simpler and less capable. And honestly I probably wouldn’t have agreed to take on the job in the first place.
- lproven 2mo ago
  > far smaller, simpler and less capable So, better, in other words?
  - bensyverson 2mo ago
    Sure, in the same way that nano is better than Microsoft Word. Unfortunately, sometimes nano is not enough. In the case of this client project, we are only now getting to enough functionality to be generally useful. The complexity that exists in the project is necessary complexity. A lot of coders are simply wrong about complexity, and feel it can always be reduced.
    - 2mo ago

iammrpayments 2mo ago

Not the 1000x more productive numbers again please. Is this a meme or something? If you said +17% I might have taken you seriously.

socalgal2 2mo ago
I gave the 1000x example because a single semi-truck can generally carry 240x what a horse can carry. A horse can carry 150-200lbs. A semi-truck can carry 42,000 to 48,000 pounds. The largest truck carries 2.2 million pounds or 11000x the horse. To see an example of this 1000x, all we have to do is look at the flood of bug reports and fixes to browsers and OSes. Does it do 1000x for every project? No, and neither does a pickup truck do 1000x what a horse does. But, even a small pickup truck does more than a horse for most use cases (carrying people or stuff) and similarly, LLMs do more for most use cases.

lproven 2mo ago

> Obviously you are correct about "how you feel". I posted the link. I didn't write it. My name is Liam Proven, thus "lproven", while they are Joshua Barretto, thus "jsbarretto".