6 ms·

14% - that's the first time i've seen a hard number like that. It's significant. Have there been any good studies on programmer productivity? 14% would be a ju

14% - that's the first time i've seen a hard number like that. It's significant. Have there been any good studies on programmer productivity? 14% would be a just a blip in the constant flow of productivity-enhancing language, tools, architectures etc. that have been steady accelerating the pace and scale of development for decades ... When I moved from Java to Rails, for example ... definitely at least 14% more productive, I'm guessing much much higher. Even just from load times. Or going from printed documentation all the way to Stack Overflow - how many multiples was that? 68000 to M2 ... page reload to React's fast refresh ... etc. etc.

masswerk 3y ago
> 14% - that's the first time i've seen a hard number like that. It's significant. On the other hand, "zero or small negative effects on the most experienced/most able workers". In other words, if you're skilled, there's nothing AI can do for you…
- adamsmith143 3y ago
  >In other words, if you're skilled, there's nothing AI can do for you… Yet...
- hugozap 3y ago
  The study group were mostly customer support agents. The nature of that work is very different from other knowledge workers so probably the results don't map to more creative fields.
- tsunamifury 3y ago
  No, it’s much darker. It means if you’re skilled the gap between you and unskilled just got much much smaller. Think about that…
  - qup 3y ago
    That's awesome. Let's make progress. Their success is not your downfall.
    - tsunamifury 3y ago
      I really love your optimism here, but as an ex-executive I can see how this will be used -- to put wage pressure on any skilled/senior leaders by inflowing a churn of unskilled to replace them. I don't think you grasp how this will play out over multi-game
      - scrollaway 3y ago
        Different companies and executives will play this out differently. Some will replace their unskilled workers, some will fire everyone, some will ignore AI altogether. What matters is which of these strategies will result in actual success. This is honestly far too hard to tell at this point.
        tsunamifury 3y ago
        You can apply the law of "power wins" I think to this pretty well. It will be initially be used to put labor in a lower position and expand ownership margins -- until that creates an opposing power scenario. Which maybe be sooner... or could be later. Hopefully before robots become real, because otherwise it will be never. I'll say here, I've spoken to and spent time with a few well known billionaires and I'd say deep down, they are exterminists no matter how nice a face they try to put on it. Over time they just come to believe most people don't matter at all. Its really dark, and the sooner we come to terms with that the better.
        visarga 3y ago
        Software has been cannibalising itself for 70 years, and yet look and behold - being a programmer is still one of the best paid jobs. I don't believe a simple 20-50% boost for intermediate users is going to change things much.
  - jiggawatts 3y ago
    That’s what Google did, supposedly for “everyone”. In practice, my coworkers call me to find and then explain solutions to them. I don’t see this changing with AIs. Having a “super Google” is great… for people already comfortable with such tools and capable of using them. My coworkers will now ask me to ask the AI… on their… behalf. OMG! I’m going to turn into Lieutenant Tawny Madison: https://chat.openai.com/share/0efbca4d-9e33-4af0-9ed6-2f8821c57201 https://chat.openai.com/share/0efbca4d-9e33-4af0-9ed6-2f8821...
  - masswerk 3y ago
    That's the way with about any real leap in productivity applications. About every time you add "smart" to something, you take from skilled domain experts. Just think of when Photoshop 4.0 added layers and now composing was for everyone, not just for those venturing into channel operations… (To be fair, here, things got much easier even for those who managed previously without this.)
- jonnycat 3y ago
  I agree with the overall assessment, but there's a catch: even "highly skilled" workers are not skilled in "all the things". I've felt that ChatGPT hasn't increased my productivity in my core languages and frameworks, but it has helped immensely in areas where I'm not an expert. To be concrete about it, I recently ported a project from SwiftUI on iOS (an ecosystem I am very comfortable with) to an Android app written in Kotlin (an ecosystem I mostly dread). I don't find ChatGPT very helpful with my day-to-day Swift stuff, but it was incredibly helpful with the Android work, from language syntax to idioms to outright translation of code.
  - cudgy 3y ago
    What framework or approach did it recommend you use for the Android UI? Compose? XML layouts? From code? I tried using the Bing chat interface, and it repeatedly pushed me in the direction of using Microsoft tools to accomplish tasks for a cross-compile mid tier solution. I had to explicitly tell it to exclude them.
  - rvieira 3y ago
    That's exactly my experience. In areas where I'm an expert, I find myself correcting ChatGPT constantly. But for a language I want to learn, it's been really helpful to get me started on a simple algorithm or debug an error message.
  - dgroshev 3y ago
    I can't help but think that it sounds like productised Gell-Mann Amnesia. If a system is useless and/or counterproductive in areas you're an expert in, but appears useful in areas that you aren't, shouldn't it be a red flag? How would you know if solutions it comes up with are bad, wrong, or just bad practices?
- visarga 3y ago
  > In other words, if you're skilled, there's nothing AI can do for you… If the task you are working on is phone support. If you are a dev, you are perpetually learning new things, it's not possible to memorise the whole field so AI would have more opportunity to help.
- deleted 3y ago
  [deleted]
- photochemsyn 3y ago
  That's mostly how I interpreted it as well: > "Customer support agents using an AI tool to guide their conversations saw a nearly 14 percent increase in [overall] productivity, with 35 percent improvements for the lowest skilled and least experienced workers, and zero or small negative effects on the most experienced/most able workers... [out of] 5,000 agents working for a Fortune 500 software company." AI quality is somewhere in the middle between highly skilled and neophyte (at present anyway).
nkingsy 3y ago
The 35% improvement for low skill workers and 0% improvement for high skill workers seems to confirm everyone’s worst fears about this stuff.
- adamsmith143 3y ago
  Why would this be scary? You've upgraded the productivity of maybe hundreds of millions of people.
  - jasmer 3y ago
    It's scary because AI is going to put the developing world out of business. This coming century is supposed to be about 'Rest of World' - first China, then India and the rest. But it's going to risk the ability of the nearly-unskilled labour to help.
    - RecycledEle 3y ago
      The Developing World will have ChatGPT as well. They can learn new things and cover over the things they do now know. Hopefully they can join the Developed World quickly. Every time someone says X is going to suffer under ChatGPT, I remind that person that X will also improve their productivity by using ChatGPT. ChatGPT is the most amazing personal tutor I can find. I'm adding 1 IT cert per month, largely due to ChatGPT's help as my personal tutor.
      - deleted 3y ago
        [deleted]
      - jasmer 3y ago
        My gosh 'the robots that replaced the factory workers ... will just help the workers - so now worries!' Do you know what happened to manufacturing when automation low-cost outsourcing happened? It was wiped out. It was devastating for certain sectors of the economy, even as 'net productivity rose' - the surpluses were acquired by some, not others. A lot of 'ghettos' in the US are a direct result of mass factory closures. Now - imagine that happening over the developing world, or rather, factories that were supposed to open, never did. The developing world are 'services export' economy, with things like call centres etc. - and AI will more likely than not just evaporate those roles. That those people will have 'access to ChatGPT' is besides the point when most of them don't even have computers (just mobile phones), or any way to apply that knowledge. It's a bit like saying: 'The developing world has access to Wikipedia and all of Harvard courses online! They should all have great jobs!' Unfortunately that's not how it works. AI is going to help white collar workers, not pink or blue collar work which is low-skilled.
NathanFulton 3y ago
> Have there been any good studies on programmer productivity? Yes, but mostly from the companies developing these products: * The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot". (Grep for "but speed is important, too" in [1].) * Amazon's Q1 earnings report [3] includes a paragraph stating that "participants [using] CodeWhisperer completed tasks 57% faster (on average) and were 27% more likely to complete them successfully than those who didn’t use CodeWhisperer. * I seem to remember seeing something on Replit's blog a while back with a similar number, but can't find it anymore, so maybe I'm mistaken. These speed numbers are on specific and well-specified programming tasks, which is of course only one part of a developer's job, so there's a lot of opportunity for impactful research in this space. I suspect that the Empirical Software Engineering community will be awash in third-party empirical studies asking all sorts of more specific productivity questions by this time next year. [1] https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ https://github.blog/2022-09-07-research-quantifying-github-c... [2] https://arxiv.org/pdf/2302.06590.pdf https://arxiv.org/pdf/2302.06590.pdf [3] https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/Q1-2023-Amazon-Earnings-Release.pdf https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/...
jacquesm 3y ago
My biggest jump in productivity comes when I switch off the internet router.
petercooper 3y ago
I feel like it's making me somewhat more productive because of how it gives me a kick in the butt and stops my mind wandering off. Getting immediate suggestions and ideas from a "third party" on whatever I'm doing leads me on to the next step of whatever I'm doing rather than getting tempted to procrastinate. I doubt this will benefit everyone, but I guess this is how pair programming can be quite effective for some people too.
- sitkack 3y ago
  GenAI allows me to stay in a flow state for longer and it reduces cognitive fatigue.
  - Smaug123 3y ago
    I guess you're using ChatGPT 3.5? I'd have thought v4 is simply too slow and you'd be unavoidably jolted out of flow waiting for it.
    - sitkack 3y ago
      Both. v4 is fast during periods of low load, it is slower now than when I first got access. My usage patterns have shifted because of this, so I write longer prompts, have it do more work and switch to different tab. I'd pay more than twice as much to have it be at least 50% faster.
- ardrak 3y ago
  I feel it too, even if the suggestion is wrong it still trigger my brain to think "No, it should be like this." and then I write the thing. Rinse and repeat.
- ghaff 3y ago
  I've written a few things (words) where one of the LLMs gave me a pretty good starting point. It needed some work but it saved me Googling around to get some sort of starting draft down on paper--which invariably turns into more Googling around and distractions. It's always been stuff I've known well enough that I knew what was right, wrong, and insipid. But it was a starting point (which can be useful, especially if you're just going for workmanlike anyway).
visarga 3y ago
Probably VSCode is already more than 14% boost even without Copilot
nbashaw 3y ago
Even more significant when you take this into account: > The tool was rolled out to the agents gradually, mostly between November 2020 and February 2021. So the 14% gain came from vastly less capable systems than what we now have today.
shane_b 3y ago
I don’t know where people are getting such low numbers. I’ve been a developer for 10 years. I care a lot about optimizing my workflow. When copilot was released, I’d say I got a 15% increase. When ChatGPT was released it was like 50% at least and I can’t imagine going back. I remember how slow it was now. My advice would be to force yourself to leverage it more or something. I hate googling now. I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing. How are y’all using it?
- marstall 3y ago
  curious to hear more about the kind of projects you're working on and the kind of problems it helps you with. > I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing. can you explain more about this I don't quite understand this passage.

masswerk 3y ago

> 14% - that's the first time i've seen a hard number like that. It's significant. On the other hand, "zero or small negative effects on the most experienced/most able workers". In other words, if you're skilled, there's nothing AI can do for you…

adamsmith143 3y ago
>In other words, if you're skilled, there's nothing AI can do for you… Yet...
hugozap 3y ago
The study group were mostly customer support agents. The nature of that work is very different from other knowledge workers so probably the results don't map to more creative fields.
tsunamifury 3y ago
No, it’s much darker. It means if you’re skilled the gap between you and unskilled just got much much smaller. Think about that…
- qup 3y ago
  That's awesome. Let's make progress. Their success is not your downfall.
  - tsunamifury 3y ago
    I really love your optimism here, but as an ex-executive I can see how this will be used -- to put wage pressure on any skilled/senior leaders by inflowing a churn of unskilled to replace them. I don't think you grasp how this will play out over multi-game
    - scrollaway 3y ago
      Different companies and executives will play this out differently. Some will replace their unskilled workers, some will fire everyone, some will ignore AI altogether. What matters is which of these strategies will result in actual success. This is honestly far too hard to tell at this point.
      - tsunamifury 3y ago
        You can apply the law of "power wins" I think to this pretty well. It will be initially be used to put labor in a lower position and expand ownership margins -- until that creates an opposing power scenario. Which maybe be sooner... or could be later. Hopefully before robots become real, because otherwise it will be never. I'll say here, I've spoken to and spent time with a few well known billionaires and I'd say deep down, they are exterminists no matter how nice a face they try to put on it. Over time they just come to believe most people don't matter at all. Its really dark, and the sooner we come to terms with that the better.
        visarga 3y ago
        Software has been cannibalising itself for 70 years, and yet look and behold - being a programmer is still one of the best paid jobs. I don't believe a simple 20-50% boost for intermediate users is going to change things much.
- jiggawatts 3y ago
  That’s what Google did, supposedly for “everyone”. In practice, my coworkers call me to find and then explain solutions to them. I don’t see this changing with AIs. Having a “super Google” is great… for people already comfortable with such tools and capable of using them. My coworkers will now ask me to ask the AI… on their… behalf. OMG! I’m going to turn into Lieutenant Tawny Madison: https://chat.openai.com/share/0efbca4d-9e33-4af0-9ed6-2f8821c57201 https://chat.openai.com/share/0efbca4d-9e33-4af0-9ed6-2f8821...
- masswerk 3y ago
  That's the way with about any real leap in productivity applications. About every time you add "smart" to something, you take from skilled domain experts. Just think of when Photoshop 4.0 added layers and now composing was for everyone, not just for those venturing into channel operations… (To be fair, here, things got much easier even for those who managed previously without this.)
jonnycat 3y ago
I agree with the overall assessment, but there's a catch: even "highly skilled" workers are not skilled in "all the things". I've felt that ChatGPT hasn't increased my productivity in my core languages and frameworks, but it has helped immensely in areas where I'm not an expert. To be concrete about it, I recently ported a project from SwiftUI on iOS (an ecosystem I am very comfortable with) to an Android app written in Kotlin (an ecosystem I mostly dread). I don't find ChatGPT very helpful with my day-to-day Swift stuff, but it was incredibly helpful with the Android work, from language syntax to idioms to outright translation of code.
- cudgy 3y ago
  What framework or approach did it recommend you use for the Android UI? Compose? XML layouts? From code? I tried using the Bing chat interface, and it repeatedly pushed me in the direction of using Microsoft tools to accomplish tasks for a cross-compile mid tier solution. I had to explicitly tell it to exclude them.
- rvieira 3y ago
  That's exactly my experience. In areas where I'm an expert, I find myself correcting ChatGPT constantly. But for a language I want to learn, it's been really helpful to get me started on a simple algorithm or debug an error message.
- dgroshev 3y ago
  I can't help but think that it sounds like productised Gell-Mann Amnesia. If a system is useless and/or counterproductive in areas you're an expert in, but appears useful in areas that you aren't, shouldn't it be a red flag? How would you know if solutions it comes up with are bad, wrong, or just bad practices?
visarga 3y ago
> In other words, if you're skilled, there's nothing AI can do for you… If the task you are working on is phone support. If you are a dev, you are perpetually learning new things, it's not possible to memorise the whole field so AI would have more opportunity to help.
deleted 3y ago
[deleted]
photochemsyn 3y ago
That's mostly how I interpreted it as well: > "Customer support agents using an AI tool to guide their conversations saw a nearly 14 percent increase in [overall] productivity, with 35 percent improvements for the lowest skilled and least experienced workers, and zero or small negative effects on the most experienced/most able workers... [out of] 5,000 agents working for a Fortune 500 software company." AI quality is somewhere in the middle between highly skilled and neophyte (at present anyway).

nkingsy 3y ago

The 35% improvement for low skill workers and 0% improvement for high skill workers seems to confirm everyone’s worst fears about this stuff.

adamsmith143 3y ago
Why would this be scary? You've upgraded the productivity of maybe hundreds of millions of people.
- jasmer 3y ago
  It's scary because AI is going to put the developing world out of business. This coming century is supposed to be about 'Rest of World' - first China, then India and the rest. But it's going to risk the ability of the nearly-unskilled labour to help.
  - RecycledEle 3y ago
    The Developing World will have ChatGPT as well. They can learn new things and cover over the things they do now know. Hopefully they can join the Developed World quickly. Every time someone says X is going to suffer under ChatGPT, I remind that person that X will also improve their productivity by using ChatGPT. ChatGPT is the most amazing personal tutor I can find. I'm adding 1 IT cert per month, largely due to ChatGPT's help as my personal tutor.
    - deleted 3y ago
      [deleted]
    - jasmer 3y ago
      My gosh 'the robots that replaced the factory workers ... will just help the workers - so now worries!' Do you know what happened to manufacturing when automation low-cost outsourcing happened? It was wiped out. It was devastating for certain sectors of the economy, even as 'net productivity rose' - the surpluses were acquired by some, not others. A lot of 'ghettos' in the US are a direct result of mass factory closures. Now - imagine that happening over the developing world, or rather, factories that were supposed to open, never did. The developing world are 'services export' economy, with things like call centres etc. - and AI will more likely than not just evaporate those roles. That those people will have 'access to ChatGPT' is besides the point when most of them don't even have computers (just mobile phones), or any way to apply that knowledge. It's a bit like saying: 'The developing world has access to Wikipedia and all of Harvard courses online! They should all have great jobs!' Unfortunately that's not how it works. AI is going to help white collar workers, not pink or blue collar work which is low-skilled.

NathanFulton 3y ago

> Have there been any good studies on programmer productivity? Yes, but mostly from the companies developing these products: * The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find "developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot". (Grep for "but speed is important, too" in [1].) * Amazon's Q1 earnings report [3] includes a paragraph stating that "participants [using] CodeWhisperer completed tasks 57% faster (on average) and were 27% more likely to complete them successfully than those who didn’t use CodeWhisperer. * I seem to remember seeing something on Replit's blog a while back with a similar number, but can't find it anymore, so maybe I'm mistaken. These speed numbers are on specific and well-specified programming tasks, which is of course only one part of a developer's job, so there's a lot of opportunity for impactful research in this space. I suspect that the Empirical Software Engineering community will be awash in third-party empirical studies asking all sorts of more specific productivity questions by this time next year. [1] https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ https://github.blog/2022-09-07-research-quantifying-github-c... [2] https://arxiv.org/pdf/2302.06590.pdf https://arxiv.org/pdf/2302.06590.pdf [3] https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/Q1-2023-Amazon-Earnings-Release.pdf https://s2.q4cdn.com/299287126/files/doc_financials/2023/q1/...

jacquesm 3y ago

My biggest jump in productivity comes when I switch off the internet router.

petercooper 3y ago

I feel like it's making me somewhat more productive because of how it gives me a kick in the butt and stops my mind wandering off. Getting immediate suggestions and ideas from a "third party" on whatever I'm doing leads me on to the next step of whatever I'm doing rather than getting tempted to procrastinate. I doubt this will benefit everyone, but I guess this is how pair programming can be quite effective for some people too.

sitkack 3y ago
GenAI allows me to stay in a flow state for longer and it reduces cognitive fatigue.
- Smaug123 3y ago
  I guess you're using ChatGPT 3.5? I'd have thought v4 is simply too slow and you'd be unavoidably jolted out of flow waiting for it.
  - sitkack 3y ago
    Both. v4 is fast during periods of low load, it is slower now than when I first got access. My usage patterns have shifted because of this, so I write longer prompts, have it do more work and switch to different tab. I'd pay more than twice as much to have it be at least 50% faster.
ardrak 3y ago
I feel it too, even if the suggestion is wrong it still trigger my brain to think "No, it should be like this." and then I write the thing. Rinse and repeat.
ghaff 3y ago
I've written a few things (words) where one of the LLMs gave me a pretty good starting point. It needed some work but it saved me Googling around to get some sort of starting draft down on paper--which invariably turns into more Googling around and distractions. It's always been stuff I've known well enough that I knew what was right, wrong, and insipid. But it was a starting point (which can be useful, especially if you're just going for workmanlike anyway).

visarga 3y ago

Probably VSCode is already more than 14% boost even without Copilot

nbashaw 3y ago

Even more significant when you take this into account: > The tool was rolled out to the agents gradually, mostly between November 2020 and February 2021. So the 14% gain came from vastly less capable systems than what we now have today.

shane_b 3y ago

I don’t know where people are getting such low numbers. I’ve been a developer for 10 years. I care a lot about optimizing my workflow. When copilot was released, I’d say I got a 15% increase. When ChatGPT was released it was like 50% at least and I can’t imagine going back. I remember how slow it was now. My advice would be to force yourself to leverage it more or something. I hate googling now. I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing. How are y’all using it?

marstall 3y ago
curious to hear more about the kind of projects you're working on and the kind of problems it helps you with. > I’ll find a page, copy the entire thing into gpt4 and the file i’m using with the error message and i have to do nothing. can you explain more about this I don't quite understand this passage.