4 ms·

I'm sure I've read somewhere - and it annoys me immensely that I can't recall the source - that SWEs perceive they are more productive with AI, but the measurem

I'm sure I've read somewhere - and it annoys me immensely that I can't recall the source - that SWEs perceive they are more productive with AI, but the measurements say they aren't.

tartoran 2y ago
More productive with AI or more productive in general?
bwestergard 2y ago
This seems right, intuitively, but I'd love to see a source. I've noticed that when I detect chatbot-style code in a coworker's PR, I often find subtle problems. But it's harder for me to spot the problems in code I got out of a chatbot because I am primed by the very activity of writing the prompt to see what I desire in the output.
bluefirebrand 2y ago
This is my observation. Not a scientific measurement by any means but from what I can see it isn't speeding anyone up If I had to guess, people feel more productive because they are doing less of the work they are used to and more review / testing, but to reach the same level of confidence the review takes much longer And the people who are not doing thorough review are producing absolute garbage, and are basically clueless
h4ny 2y ago
Tangentially related, I feel that SWEs who claim that they are more productive with AI haven't actually demonstrated with real examples of how they are actually more productive. Nobody I follow (including some prominent bloggers and YouTubers) claiming productivity increase is recording or detailing any workflow or showing real world, non-hobby (scalable, maintainable, readable, secure, etc.) workflows of how to do it. It's like everyone who "knows what they are doing" is hiding what the secret sauce for a competitive edge or that they are all just mediocre SWEs hyping AI up and lying because it makes them more money. Even real SWEs in large companies I know can't really seem to tell me how their productivity is increasing, and when you dig deeper it always seem to be well-scoped and well-understood problems (which is great, but doesn't match the level of hype and productivity increase that everyone else is claiming) -- and they still have to be very careful with reviewing (for now). It's almost like AI makes SWE brains go mush and forget about logic and data.
- simonw 2y ago
  I wrote 4,800 words about how I'm using LLMs to help me code here, because I was frustrated at how little detailed information there was on that topic: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
- nerdponx 2y ago
  I'm a "data scientist", but I have absolutely improved my productivity in the last year or so by conversing with LLM chatbots to work through tough problems, get ideas, figure out project plans, etc. I can see the effect in my list of completed projects, the overall speed isn't that much higher, but the quality has definitely gone up, because I'm able to work through things more quickly and get to good solutions faster, so I can spend the more time iterating on good ideas and less time trying figure out which ideas are even good. For programming, meh, it helps when I'm really tired and don't want to read documentation. Can't imagine using it in a serious capacity for writing code except in a huge codebase, where I might want it to explain to me how things fit together in order to make some change or fix a bug.
- MattSayar 2y ago
  In fairness, it's also still a NEW technology in the scale of tech. For comparison, it takes years after a gaming console is released for teams to optimize and squeeze every last ounce of performance out of the hardware. We're just getting started with AI, and we're still "stuck" in the chat interfaces because of the storming success of ChatGPT a few years ago. Cursor, GitHub Copilot etc. are cool but they're still "launch titles" to continue my analogy from above. New models are still coming out (but slowing down) with increased capabilities, context windows, etc. and I'm sure the killer app is still waiting to be unearthed. In the meantime, I'm having a lot of fun building my hobby code. Collectively, we're going to morph that into something more scalable and enterprisey, it's just a matter of time.
ohgr 2y ago
It depends how you measure productivity and value. And who is measuring it. And who tells the story. If the developer writes 6,000 lines of utter dog shit with AI that causes your customers to leave, well.
abalashov 2y ago
Interestingly, this is the conclusion reached by the major militaries, Axis and Allied alike, at the end of extensive experiments with amphetamines in WWII. They certainly made pilots and soldiers feel more confident, engaged and attentive, but the quality of the output was at best unchanged and at worst markedly inferior.
- fc417fc802 2y ago
  Depending on how they're used. That's a pretty big caveat. You can't replace sleep with them and expect the same performance. They're still quite useful though.
bobbiechen 2y ago
I can believe that, with personal experience from a non-AI tool! A few years back, I wrote a puzzle solving tool (semaphore decoder) that felt faster than using a lookup table manually, but was actually very similar in time. Those notes: https://bobbiechen.com/blog/2020/5/28/the-making-of-semaphore-decoder https://bobbiechen.com/blog/2020/5/28/the-making-of-semaphor... Regardless of the speed, it certainly felt easier because I didn't have to think as hard, and maybe that extra freshness would improve productivity for later tasks. I wonder if there's any effect like that for AI coding tools - it makes you happier to be less tired.
spacemadness 2y ago
I perceive quite the opposite. Rarely do I see it producing workable solutions and it often just creates noise. What’s worse is the mistakes it makes sometimes are nuanced, and not the kind of mistakes a human coder would make, causing me to waste a lot of time finding the mistake. I think it’s more useful to get ideas from, or treat it like a trainer when learning a new language, but code generation seems really poor to me still. The only ones I see arguing that its not the case are junior coders making slop apps that do nothing all that interesting.
nyarlathotep_ 2y ago
My experience is that "the rent ends up due" basically you "pay cognitively" up front (building an understanding from/while doing) or later (when you have to troubleshoot something in a largely LLM-generated tangle. Basically it moves from--"oh yeah I wrote something with this schema earlier" to "I saw some DB code fly out around an hour ago; maybe it's there. Where was it? `grep models ./src` wait, was it in `db` and other silly stuff like that. No free lunches or whatever. I'm not an extensive LLM user for programming and remain mostly agnostic on overall uses for development (sure a brand-new React thing and you're sailing, but a huge old crusty codebase, even in a language well-represented in the training set is a LOT less promising IME) However there's use cases where I'd straight up not do whatever until the latest minute possible that I use LLMs for now: Cloudformation, various utility bash scripts, simple AWS Lambda functions and other things I consider annoying chores. For me, these cases alone have been an unambiguous victory.

I'm sure I've read somewhere - and it annoys me immensely that I can't recall the source - that SWEs perceive they are more productive with AI, but the measurements say they aren't.

tartoran 2y ago

More productive with AI or more productive in general?

bwestergard 2y ago

This seems right, intuitively, but I'd love to see a source. I've noticed that when I detect chatbot-style code in a coworker's PR, I often find subtle problems. But it's harder for me to spot the problems in code I got out of a chatbot because I am primed by the very activity of writing the prompt to see what I desire in the output.

bluefirebrand 2y ago

This is my observation. Not a scientific measurement by any means but from what I can see it isn't speeding anyone up If I had to guess, people feel more productive because they are doing less of the work they are used to and more review / testing, but to reach the same level of confidence the review takes much longer And the people who are not doing thorough review are producing absolute garbage, and are basically clueless

h4ny 2y ago

Tangentially related, I feel that SWEs who claim that they are more productive with AI haven't actually demonstrated with real examples of how they are actually more productive. Nobody I follow (including some prominent bloggers and YouTubers) claiming productivity increase is recording or detailing any workflow or showing real world, non-hobby (scalable, maintainable, readable, secure, etc.) workflows of how to do it. It's like everyone who "knows what they are doing" is hiding what the secret sauce for a competitive edge or that they are all just mediocre SWEs hyping AI up and lying because it makes them more money. Even real SWEs in large companies I know can't really seem to tell me how their productivity is increasing, and when you dig deeper it always seem to be well-scoped and well-understood problems (which is great, but doesn't match the level of hype and productivity increase that everyone else is claiming) -- and they still have to be very careful with reviewing (for now). It's almost like AI makes SWE brains go mush and forget about logic and data.

simonw 2y ago
I wrote 4,800 words about how I'm using LLMs to help me code here, because I was frustrated at how little detailed information there was on that topic: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
nerdponx 2y ago
I'm a "data scientist", but I have absolutely improved my productivity in the last year or so by conversing with LLM chatbots to work through tough problems, get ideas, figure out project plans, etc. I can see the effect in my list of completed projects, the overall speed isn't that much higher, but the quality has definitely gone up, because I'm able to work through things more quickly and get to good solutions faster, so I can spend the more time iterating on good ideas and less time trying figure out which ideas are even good. For programming, meh, it helps when I'm really tired and don't want to read documentation. Can't imagine using it in a serious capacity for writing code except in a huge codebase, where I might want it to explain to me how things fit together in order to make some change or fix a bug.
MattSayar 2y ago
In fairness, it's also still a NEW technology in the scale of tech. For comparison, it takes years after a gaming console is released for teams to optimize and squeeze every last ounce of performance out of the hardware. We're just getting started with AI, and we're still "stuck" in the chat interfaces because of the storming success of ChatGPT a few years ago. Cursor, GitHub Copilot etc. are cool but they're still "launch titles" to continue my analogy from above. New models are still coming out (but slowing down) with increased capabilities, context windows, etc. and I'm sure the killer app is still waiting to be unearthed. In the meantime, I'm having a lot of fun building my hobby code. Collectively, we're going to morph that into something more scalable and enterprisey, it's just a matter of time.

ohgr 2y ago

It depends how you measure productivity and value. And who is measuring it. And who tells the story. If the developer writes 6,000 lines of utter dog shit with AI that causes your customers to leave, well.

abalashov 2y ago

Interestingly, this is the conclusion reached by the major militaries, Axis and Allied alike, at the end of extensive experiments with amphetamines in WWII. They certainly made pilots and soldiers feel more confident, engaged and attentive, but the quality of the output was at best unchanged and at worst markedly inferior.

fc417fc802 2y ago
Depending on how they're used. That's a pretty big caveat. You can't replace sleep with them and expect the same performance. They're still quite useful though.

bobbiechen 2y ago

I can believe that, with personal experience from a non-AI tool! A few years back, I wrote a puzzle solving tool (semaphore decoder) that felt faster than using a lookup table manually, but was actually very similar in time. Those notes: https://bobbiechen.com/blog/2020/5/28/the-making-of-semaphore-decoder https://bobbiechen.com/blog/2020/5/28/the-making-of-semaphor... Regardless of the speed, it certainly felt easier because I didn't have to think as hard, and maybe that extra freshness would improve productivity for later tasks. I wonder if there's any effect like that for AI coding tools - it makes you happier to be less tired.

spacemadness 2y ago

I perceive quite the opposite. Rarely do I see it producing workable solutions and it often just creates noise. What’s worse is the mistakes it makes sometimes are nuanced, and not the kind of mistakes a human coder would make, causing me to waste a lot of time finding the mistake. I think it’s more useful to get ideas from, or treat it like a trainer when learning a new language, but code generation seems really poor to me still. The only ones I see arguing that its not the case are junior coders making slop apps that do nothing all that interesting.

nyarlathotep_ 2y ago

My experience is that "the rent ends up due" basically you "pay cognitively" up front (building an understanding from/while doing) or later (when you have to troubleshoot something in a largely LLM-generated tangle. Basically it moves from--"oh yeah I wrote something with this schema earlier" to "I saw some DB code fly out around an hour ago; maybe it's there. Where was it? `grep models ./src` wait, was it in `db` and other silly stuff like that. No free lunches or whatever. I'm not an extensive LLM user for programming and remain mostly agnostic on overall uses for development (sure a brand-new React thing and you're sailing, but a huge old crusty codebase, even in a language well-represented in the training set is a LOT less promising IME) However there's use cases where I'd straight up not do whatever until the latest minute possible that I use LLMs for now: Cloudformation, various utility bash scripts, simple AWS Lambda functions and other things I consider annoying chores. For me, these cases alone have been an unambiguous victory.