4 ms·

I feel like HN is quite divided about that actually, A couple of days I started a survey which I plan to run monthly to see how the community feels about "LLM p

I feel like HN is quite divided about that actually, A couple of days I started a survey which I plan to run monthly to see how the community feels about "LLM productivity etc". Now I have ~250 answers, need a couple more to make it significant but as of now it looks like >90% report productivity gains from AI tools - happy if you participate, only takes a minute: https://agentic-coding-survey.pages.dev/ https://agentic-coding-survey.pages.dev/

anonymous908213 9mo ago
Note that self-reporting productivity gains is a completely unreliable and unscientific metric. One study[1], small in scope but a noteworthy data point, found that over the course of the study that LLMs reduced productivity by ~20% but even after the fact the participants felt that on average their productivity had increased by ~20%. This study is surely not the end-all be-all and you could find ways to criticise it or say it doesn't apply or they were doing it wrong or whatever reason you think the developers should have had increased productivity, but the point is that people cannot accurately judge their own productivity by vibes alone. [1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
- 7777777phil 9mo ago
  If you look at the survey it's not only about productivity it's also about usage, model choice etc. But I agree with you self reported productivity gains is to be taken with a grain of salt. But then what else would you propose? The goal is to not only rely on benchmarks for model performance but develop some kind of TIOBE Index for LLMs.
  - direwolf20 9mo ago
    The ever-present rebuttal to all LLM failure anecdotes: you're using the wrong model, you're prompting it wrong, etc. All failures are always the user's fault. It couldn't possibly be that the tool is bad.
    - peteforde 9mo ago
      Of course, your logic could also be equally allied to the opposite position. Quite a few of us are tired of being told that we're imagining doing what used to take weeks multiple times in an evening.
      - anonymous908213 9mo ago
        If it generated something that saved you weeks, I think it's almost certainly because it was used for something you have absolutely zero domain understanding for and would have had to study from scratch. And I, at least, repeatedly do note that LLMs lower the barrier to entry for making proof-of-concepts. But the problem is that (1) people treat that instant gratification as a form of productivity that can replace software engineers. At most, it can make something extremely rough that is suited to one individual's very specific use case, where you mostly work around the plentiful bugs by knowing the landmines are there and not doing the behaviour that trips them; and (2) people spam these low-effort proof-of-concepts, which have no value to other people on account of how rough and lacking in ability to be extended to cover more than one person's use case they are, and this drowns out the content people actually put effort into. LLMs, when used like this, do not increase productivity on making software worth sharing with other people. While they can knock out the proof-of-concept, they cannot build it into something valuable to anyone but the prompter, and by shortcircuiting the learning process, you do not learn the skills necessary to build upon the domain yourself, meaning you still have to spend weeks learning those skills if you actually want to build something meaningful. At least this is true for everything I have observed out of the vibe-coding bubble thus far, and my own extensive experiences trying to discover the 10x boost I am told exists. I am open to being shown something genuinely great that an LLM generated in an evening if you wish to share evidence to the contrary. There is also the question of the provenance of the code, of course. Could you have saved those weeks by simply using a library? Is the LLM saving you weeks by writing the library ""from scratch"", in actuality regurgitating code from an existing library one prompt at a time? If the LLM's productivity gain is that it normalized copying and pasting open-source code wholesale while calling it your own, I don't think that's the great advancement for humanity it is portrayed as.
illegalbyte2 9mo ago
Really like the ux of that survey - super easy to fill out, is it just a custom web form or you used a library?
- 7777777phil 9mo ago
  Yes exactly, it's a standalone cloudflare page with some custom html/css that writes to a D1 (Cloudflare SQL DB) for results and rate limits, thats's it. I looked at so many survey tools but none offered what I was looking for (simple single page form, no email, no signup, no tracking) so I built this (with claude) Thanks for the feedback!

anonymous908213 9mo ago

Note that self-reporting productivity gains is a completely unreliable and unscientific metric. One study[1], small in scope but a noteworthy data point, found that over the course of the study that LLMs reduced productivity by ~20% but even after the fact the participants felt that on average their productivity had increased by ~20%. This study is surely not the end-all be-all and you could find ways to criticise it or say it doesn't apply or they were doing it wrong or whatever reason you think the developers should have had increased productivity, but the point is that people cannot accurately judge their own productivity by vibes alone. [1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

7777777phil 9mo ago
If you look at the survey it's not only about productivity it's also about usage, model choice etc. But I agree with you self reported productivity gains is to be taken with a grain of salt. But then what else would you propose? The goal is to not only rely on benchmarks for model performance but develop some kind of TIOBE Index for LLMs.
- direwolf20 9mo ago
  The ever-present rebuttal to all LLM failure anecdotes: you're using the wrong model, you're prompting it wrong, etc. All failures are always the user's fault. It couldn't possibly be that the tool is bad.
  - peteforde 9mo ago
    Of course, your logic could also be equally allied to the opposite position. Quite a few of us are tired of being told that we're imagining doing what used to take weeks multiple times in an evening.
    - anonymous908213 9mo ago
      If it generated something that saved you weeks, I think it's almost certainly because it was used for something you have absolutely zero domain understanding for and would have had to study from scratch. And I, at least, repeatedly do note that LLMs lower the barrier to entry for making proof-of-concepts. But the problem is that (1) people treat that instant gratification as a form of productivity that can replace software engineers. At most, it can make something extremely rough that is suited to one individual's very specific use case, where you mostly work around the plentiful bugs by knowing the landmines are there and not doing the behaviour that trips them; and (2) people spam these low-effort proof-of-concepts, which have no value to other people on account of how rough and lacking in ability to be extended to cover more than one person's use case they are, and this drowns out the content people actually put effort into. LLMs, when used like this, do not increase productivity on making software worth sharing with other people. While they can knock out the proof-of-concept, they cannot build it into something valuable to anyone but the prompter, and by shortcircuiting the learning process, you do not learn the skills necessary to build upon the domain yourself, meaning you still have to spend weeks learning those skills if you actually want to build something meaningful. At least this is true for everything I have observed out of the vibe-coding bubble thus far, and my own extensive experiences trying to discover the 10x boost I am told exists. I am open to being shown something genuinely great that an LLM generated in an evening if you wish to share evidence to the contrary. There is also the question of the provenance of the code, of course. Could you have saved those weeks by simply using a library? Is the LLM saving you weeks by writing the library ""from scratch"", in actuality regurgitating code from an existing library one prompt at a time? If the LLM's productivity gain is that it normalized copying and pasting open-source code wholesale while calling it your own, I don't think that's the great advancement for humanity it is portrayed as.

illegalbyte2 9mo ago

Really like the ux of that survey - super easy to fill out, is it just a custom web form or you used a library?

7777777phil 9mo ago
Yes exactly, it's a standalone cloudflare page with some custom html/css that writes to a D1 (Cloudflare SQL DB) for results and rate limits, thats's it. I looked at so many survey tools but none offered what I was looking for (simple single page form, no email, no signup, no tracking) so I built this (with claude) Thanks for the feedback!