15 ms·

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

• gibsonf1 2y ago
  Yes, but what about all the bugs created and time to debug?
• danielvaughn 2y ago
  A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.
  • itchyjunk 2y ago
    Maybe that's what the new vs experienced difference is indirectly capturing.
  • jstummbillig 2y ago
    That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.
    • tuyiown 2y ago
      Flexible, or just that only juniors have real benefits
    • infecto 2y ago
      I wish I could upvote this comment more than once. There does appear to be a prejudice with more senior programmers arguing why it cannot work, how they just cause more trouble, and other various complaints. The tools today are not perfect but they still amaze me at what is being accomplished, even a 10% gain is incredible for something that costs $10/month. I believe progress will be made in the space and the tooling in 5 years will be even better.
      • randomdata 2y ago
        > The tools today are not perfect The trouble is that they seem to be getting worse. Some time ago I was able to write an entire small application by simply providing some guidance around function names and data structures, with an LLM filling in all of the rest of the code. It worked fantastically and really showed how these tools can be a boon. I want to taste that same thrill again, but these days I'm lucky if I can get something out of it that will even compile, never mind the logical correctness. Maybe I'm just getting worse at using the tools.
      • mattgreenrocks 2y ago
        The prejudice comes down to whether they want to talk the LLM into the right solution vs applying what they know already. If you know your way around then there’s no need to go through the LLM. I think sr devs often tend to be more task focused so the idea of outsourcing the thinking to an LLM feels like another step to take on. I find Claude good at helping me find how to do things that I know are possible but I don’t have the right nomenclature for. This is an area where Google fails you, as you’re hoping someone else on the internet used similar terms as you when describing the problem. Once it spits out some sort of jargon I can latch onto, then I can Google and find docs to help. I prefer to use multiple sources vs just LLMs, partially because of hallucination, but also to keep amassing my own personal context. LLMs are excellent as librarians.
    • jncfhnb 2y ago
      My experience is juniors using them without prejudice and then not understanding why their code is wrong
      • SketchySeaBeast 2y ago
        I am seeing that a lot - juniors who can put out a lot of code but when they get stuck they can't unstick themselves, and it's hard for me to unstick them because they have a hard time walking me through what they are doing.
        • carlmr 2y ago
          I've gotten responses now on PRs of the form. "I don't know either, this is what Copilot told me." If you don't even understand your own PR, I'm not sure why you expect other people can. I have used LLMs myself, but mostly for boilerplate and one-off stuff. I think it can be quite helpful. But as soon as you stop understanding the code it generates you will create subtle bugs everywhere that will cost you dearly in the long run. I have the strong feeling that if LLMs really outsmart us to the degree that some AI gung-ho types believe, the old Kernighan quote will get a new meaning: "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." We'll be left with code nobody can debug because it was handed to us by our super smart AI that only hallucinates sometimes. We'll take the words of another AI that the code works. And then we'll hope for the best.
          • SketchySeaBeast 2y ago
            Copilot is best for me when it's letting me hit tab to auto-complete something I was going to write anyways.
            • carlmr 2y ago
              Copilot does hit that sweet to spot sometimes.
        • 2y ago
    • GoToRO 2y ago
      As a senior, you know the problem is actually finishing a project. That's the moment all those bad decisions made by junior need to be fixed. This also means that a 80% done project is more like 20% done, because in the state it is, it can not be finished: you fix one thing and break 2 more.
    • SketchySeaBeast 2y ago
      As a senior, I find that trying to use copilot really only gives me gains maybe half the time, the other half the time it leads me in the wrong direction. Googling tends to give me a better result because I can actually move through the data quicker. My belief is this is because when I need help I'm doing something uncommon or hard, as opposed to juniors who need help doing regular stuff which will have plenty of examples in the training data. I don't need help with that. It certainly has its uses - it's awesome at mocking and filling in the boilerplate unit tests.
    • marginalia_nu 2y ago
      I find their value depends a lot on what I'm doing. Anything easy I'll get insane leverage, no exaggeration I'll slap together that shit 25x faster. It's seen likely billions of lines of simple CRUD endpoints, so yeah it'll write those flawlessly for you. Anything the difficult or complex, and it's really a coinflip if it's even an advantage, most of the time it's just distracting and giving irrelevant suggestions or bad textbook-style implementations intended to demonstrate a principle but with god-awful performance. Likely because there's simply not enough training data for these types of tasks. With this in mind, I don't think it's strange that junior devs would be gushing over this and senior devs would be raising a skeptical eyebrow. Both may be correct, depending on what you work on.
      • jeremy151 2y ago
        I think for me, I'm still learning how to make these tools operate effectively. But even only a few months in, it has removed most all the annoying work and lets me concentrate on the stuff that I like. At this point, I'll often give it some context, tell it what to make and it spits out something relatively close. I look it over, call out like 10 things, each time it says "you're right to question..." and we do an iteration. After we're thru that, I tell it to write a comprehensive set of unit tests, it does that, most of them fail, it fixes them, and them we usually have something pretty solid. Once we have that base pattern, I can have it pattern and extend variants after the first solid bit of code. "Using this pattern for style and approach, make one that does XYZ instead." But what I really appreciate is, I don't have to do the plug and chug stuff. Those patterns are well defined, I'm more than happy to let the LLM do that and concentrate on steering whether it's making a wise conceptual or architectural choice. It really seems to act like a higher abstraction layer. But I think how the engineer uses the tool matters too.
    • SoftTalker 2y ago
      Juniors don't know enough to know what problems the AI code might be introducing. It might work, and the tests might pass, but it might be very fragile, full of duplicated code, unnecessary side-effects, etc. that will make future maintenance and debugging difficult. But I guess we'll be using AI for that too, so the hopefully the AI can clean up the messes that it made.
    • nuancebydefault 2y ago
      Now some junior dev can quickly make something new and fully functional in days, without knowing in detail what they are doing. As opposed to weeks by a senior originally. Personally I think that senior devs might fear a conflict within their identity. Hence they draw the 'You and the AI have no cue' card.
  • pqdbr 2y ago
    Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schema (serving the OpenAPI schema via swagger) in ~5 minutes just by prompting. Starting a project from scratch, for me at least, is rare, which probably means going back to Copilot and vanilla VSCode.
    • deleted 2y ago
      [deleted]
    • gamerDude 2y ago
      I start new projects a lot and just have a template that has everything I would need already set up for me. Do you think there is a unique value prop that AI gives you when setting up that a template would not have?
      • lherron 2y ago
        I do both. I have a basic template that also includes my specific AI instructions and conventions that are inputs for Aider/Cursor. Best of both worlds.
      • carlmr 2y ago
        AI has templates for the languages you seldom use (as long as the language is not too niche for the LLM to have trained on it).
      • pjot 2y ago
        GH Copilot chat specifically has directives like @workspace /new “scaffold a ${language} project” that automagically creates a full project structure and boilerplate. It’s been great for one off things for me at least
        • sethops1 2y ago
  • moolcool 2y ago
    I've found Copilot pretty good at removing some tedium, like it'll write docstrings pretty well most of the time, but it does almost nothing to alleviate the actual mental labour of software engineering
    • jacobsenscott 2y ago
      Removing the tedium frees your brain up to focus on the interesting parts of the problem. The "actual mental labor" is the fun part. So I like that.
      • danielvaughn 2y ago
        Yeah this has been my experience. I bucket my work largely into two categories - "creative work" and "toil". I don't want or need AI to replace my creative work, but the more it can handle toil for me, the better. As it happens, I think co-pilot is a pretty poor user experience, because it's essentially just an autocomplete, which doesn't really help me all that much, and often gets in my way. I like using Cursor with the autocomplete turned off. It gives you the option to highlight a bit of text and either refactor it with a prompt, or ask a question about it in a side chat window. That puts me in the driver seat, so to speak, so I (the user) can reach out to AI when I want to.
  • hobofan 2y ago
    I'd also expand it to "languages/frameworks that I'll never properly learn". I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing with AWK. Now with LLMs, I'm creating a lot more bash scripts, and it has gotten so easy that I'll do it for process-documentation more often. E.g. what previously was a more static step-by-step README with instructions is now accompanied with an interactive bash script that takes user input.
    • sgarland 2y ago
      While I’ll grant you that LLMs are in fact shockingly good with shell scripts, I also highly recommend shellcheck [0]. You can get it as a treesitter plugin for nvim, and I assume others, so it lints as you write. [0]: https://www.shellcheck.net/ https://www.shellcheck.net/
      • jcgrillo 2y ago
        I was just about to recommend the same, as well as shfmt and shunit, or even all 3 + a build system: https://www.pantsbuild.org/stable/docs/shell https://www.pantsbuild.org/stable/docs/shell
    • danielvaughn 2y ago
      Oh god yes, LLM for bash scripting is fantastic. Bash is one of those languages that I simply don't want to learn. Reading it makes my eyes hurt.
      • a_wild_dandan 2y ago
        You could try zx[1]! I prefer it to Bash scripting. [1] https://github.com/google/zx https://github.com/google/zx
        • 2y ago
  • empath75 2y ago
    Sounds about right to me, which is why the hysteria about AI wiping out developer jobs was always absurd. _Every_ time there has been a technology that improved developer productivity, developer jobs and pay have _increased_. There is not a limited amount of automation that can be done in the world and the cheaper and easier it gets to automate stuff, the more stuff will be economically viable to automate that wasn't before. Did IDE's eliminate developer jobs? Compilers? It's just a tool.
    • randomdata 2y ago
      The automated elevator is just a tool, but it "wiped out" the elevator operator. Which is really to say not that the elevator operator was wiped out, but that everyone became the elevator operator. Thus, by the transitive properties of supply and demand, the value of operating an elevator declined to nothing. Said hysteria was built on the same idea. After all, LLMs themselves are just compilers for a programming language that is incredibly similar to spoken language. But as the programming language is incredibly similar to the spoken language that nearly everyone already knows, the idea was that everyone would become the metaphorical elevator operator, "wiping out" programming as a job just as elevator operators were "wiped out" of a job when operating an elevator became accessible to all. The key difference, and where the hysteria is likely to fall flat, is that when riding in an elevator there isn't much else to do but be the elevator operator. You may as well do it. Your situation would not be meaningfully improved if another person was there to press the button for you. When it comes to programming, though, there is more effort involved. Even when a new programming language makes programming accessible, there remains a significant time commitment to carry out the work. The business people are still best to leave that work to the peons so they can continue to focus on the important things.
  • insane_dreamer 2y ago
    I haven't found it that useful in my main product development (which while python based, uses our own framework and therefore there's not much code for CoPilot to go on and it usually suggests methods and arguments that don't exist, which just makes extra work). Where I do find it useful are 1) questions about frameworks/languages that I don't work in much and for which there is a lot of example content (i.e., Qt, CSS); 2) very specific questions I would have done a Google Search (usually StackOverflow) for ("what's the most efficient way to CPU and RAM usage on Windows using python") - the result is pointing me to a library or some example rather than directly generating code that I can copy/paste 3) boilerplate code that I already know how to write but saves me a little time and avoids typing errors. I have the CoPilot plugin for PyCharm so I'll write it as a comment in the file and then it'll complete the next few lines. Again best results is something that is very short and specific. With anything longer I almost always have to iterate so much with CoPilot that it's not worth it anymore. 4) a quick way to search documentation Some people have said it's good at writing unit tests but I have not found that to be the case (at least not the right kind of unit tests). If I had to quantify it, I'd probably give it a 5-10% increase in productivity. Much less than I get from using a full featured IDE like PyCharm over coding in Notepad, or a really good git client over typing the git commands in the CLI. In other words, it's a productivity tool like many other tools, but I would not say it's "revolutionary".
    • nuancebydefault 2y ago
      It is revolutionary no doubt. How many pre-AI tools would accomplish an amount of 4 big features you mentioned at once?
      • insane_dreamer 2y ago
        Well, like I said, a well-designed IDE is a much bigger productivity booster than CoPilot, and I've never heard anyone describe JetBrains, NetBeans or VSCode as "revolutionary".
        • nuancebydefault 2y ago
  • rm_-rf_slash 2y ago
    AI speeds my coding up 2x-4x, depending on the breadth of requirements and complexity of new tech to implement. But coding is just a fraction of my weekly workload, and AI has been less impactful for other aspects of project management. So overall it’s 25%-50% increase in productivity.
  • jakub_g 2y ago
    I only use (free) ChatGPT sporadically, and it works best for me in areas where I'm familiar enough to call bullshit, but not familiar enough to write things myself quickly / confidently / without checking a lot of docs: - writing robust bash and using unix/macos tools - how to do X in github actions - which API endpoint do I use to do Y - synthesizing knowledge on some topic that would require dozens of browser tabs - enumerating things to consider when investigating things. Like "I'm seeing X, what could be the cause, and how I do check if it's that". For example I told it last week "git rebase is very slow, what can it be?" and it told me to use GIT_TRACE=1 which made me find a slow post-commit hook, and suggested how to skip this hook while rebasing.
    • thiht 2y ago
      Same for me. I also use it for some SQL queries involving syntax I’m unfamiliar with, like JSONB operators in Postgres. ChatGPT gives me better results, faster than Google.
• alternatex 2y ago
  I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though.
  • deleted 2y ago
    [deleted]
  • anonzzzies 2y ago
    > had to tackle after the less experienced devs have contributed their AI-driven efforts. So, like before AI then? I haven't seen AI deliver illogical nonsense that I couldn't even decipher like I have seen some outsourcing companies deliver.
    • htrp 2y ago
      AI seems to give more consistent results than outsourcing (not necessarily better, but at least more predictable failure modes)
    • trashtester 2y ago
      > that I couldn't even decipher This is one of the challenges of being a tech lead. Sometimes code is hard to comprehend. In my experience, AI delivered code is no worse than entry level developer code.
      • anonzzzies 2y ago
        But that is a HN bubble thing: I work and worked with seniors with 10-15 years under their belt who have no logic bone in their body; the worst (and an AI does not do that) is when there is a somewhat 'busy' function that has an if or switch statement and, over time, to add features or fix bugs, ifs were added. Now after 5+ years, this function is 15000 lines and is somewhat of a trained neural network adjacent thing; 100s of nested ifs, pages long, that cannot be read and cannot be followed even if you have a brain. This is made by senior staff of, usually, outsourcing companies and I have seen it very many times over the past 40 years. Not entry level, not small companies either. I know a gov tax system which was partly maintained by a very large and well known outsourcing company which has numerous of these puppies in production that no one dares to touch. AI doesn't do stuff like because it could not, which to me, is a good thing. When it gets better, it might start to do it, I don't know. People here live in a bubble where they think the world is full of people who read 'beautiful code', make tests, use git or something instead of zip$date and know how DeMorgan works; by far, most don't, not juniors, not seniors.
• wouldbecouldbe 2y ago
  I'm guiding a few and sometimes they write pretty good code with the help of GPT but then in meetings have trouble understanding and explaining things. I think it's a big productivity boost, but also a chance that the learning rate might actually be significantly slower.
• GoToRO 2y ago
  For me AI just brought back documentation. All new frameworks lack documentation big time. The last good one for me was a DOS book! I don't think newer developers even have an idea of what good documentation looks like. Even so, AI will propose different things at different times and you still need an experienced developer to make the call. In the end it replaces documentation and typing.
  • choilive 2y ago
    I think AI just made documentation more valuable. For public facing projects - your documentation just became part of the LLM's training data, so its now extra important your documentation is thorough and accurate because you will have a ton of developers getting answers from that system. For private projects, your documentation can now be fed into a finetuning dataset or a RAG system, achieving the same effect.
  • jakub_g 2y ago
    Every now and then there's a HN discussion "how do you manage internal documentation" where most commenters write something like "there's no point writing documentation because it quickly becomes outdated". (There were two such threads in just last days). Might explain why nothing is documented anymore.
    • jcgrillo 2y ago
      Maybe that also explains why (approximately) nothing really works right.
  • tensor 2y ago
    Yes, it can both help you write it but also, if you start with the documentation, e.g. the comment describing what function does, AI becomes orders of magnitude better and actually helping write the function, so it actually forces developers to better document their code!
    • Sakos 2y ago
      I've noticed the same thing. I can get great results by being detailed in my comments and/or docstrings. A lot of the time I don't want the AI actually "thinking" for itself.
• null_investor 2y ago
  They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior. I've already fixed a couple of tests like this, where people clearly used AI and didn't think about it, when in reality it was testing something wrong. Not to mention the rest of the technical debt added... looking at productivity in software development by amount of tasks is so wrong.
  • onlyrealcuzzo 2y ago
    > They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior. Let's not forget that developers some times do this, too...
  • randomdata 2y ago
    > when in reality it was testing something wrong. Must have seen AI write the implementation as well? If you're still cognizant of what you're writing on the implementation side, it's pretty hard to see a test go from failing to passing if the test is buggy. It requires you to independently introduce the same bug the LLM did, which, while not completely impossible, is unlikely. Of course, humans are prone to not understanding the requirements, and introducing what isn't really a bug in the strictest sense but rather a misfeature.
    • Jensson 2y ago
      > it's pretty hard to see a test go from failing to passing Its pretty easy to add a passing test and call it done without checking if it actually fails in the right circumstances, and then you will get a ton of buggy tests. Most developers don't do the start out at failing and then to passing ritual, especially junior ones who copies code from somewhere instead of knowing what they wrote.
• muglug 2y ago
  This is a decently-thorough study, using PRs as a productivity metric while also tracking build failures (which remained constant at MSFT but increased at Accenture). Would love to see it replicated by researchers at a company that does not have a clear financial interest in the outcome (the corresponding author here was working at Microsoft Research during the study period). > Before moving on, we discuss an additional experiment run at Accenture that was abandoned due to a large layoff affecting 42% of participants Eek
• lolinder 2y ago
  The most interesting thing about this study for me is that when they break it down by experience levels, developers who are above the median tenure show no statistically significant increase in 'productivity' (for some bad proxies of productivity), with the 95% confidence intervals actually dipping deep into the negatives on all metrics (though leaning slightly positive). This tracks with my own experience: Copilot is nice for resolving some tedium and freeing up my brain to focus more on deeper questions, but it's not as world-altering as junior devs describe it as. It's also frequently subtly wrong in ways that a newer dev wouldn't catch, which requires me to stop and tweak most things it generates in a way that a less experienced dev probably wouldn't know to. A few years into it I now have a pretty good sense for when to use Copilot and when not to—so I think it's probably a net positive for me now—but it certainly wasn't always that way. I also wonder if the possibly-decreased 'productivity' for more senior devs stems in part from the increase in 'productivity' from the juniors in the company. If the junior devs are producing more PRs that have more mistakes and take longer to review, this would potentially slow down seniors, reducing their own productivity gains proportionally.
• deleted 2y ago
  [deleted]
• sum1ren 2y ago
  When I'm using genai to write some code for me, I lose the internal mental state of what my code is doing. As such, when I do have to debug problems myself, or dream up ideas of improvements, I no longer can do this properly due to lack of internal mental state. Wonder how people who have used genai coding successfully get around this?
  • cageface 2y ago
    This is the problem I have with it. It breaks me out of the flow state that’s so important to writing good code.
  • Sakos 2y ago
    Go over the code again and again like you would if you'd written it yourself. Keep iterating on the code yourself without having the AI generate everything. In a current project I have everything internalised because I'm constantly going through the code and improving upon it myself (and using the AI to do boilerplate), even though much of it was initially generated. I'll still have the AI generate code in plenty of places, but I don't let it take over the thinking for me unless I'm unsure about something, then I ask it for possible solutions and ideas, then I go over those solutions myself.
  • LouisSayers 2y ago
    I use Claude and there's a couple things I'd suggest. 1) You need to be the boss with the AI being your assistant. You are now a project manager coming up with strict requirements of what you'd like done. Your developer (AI) needs context, constraints and needs to be told exactly what you'd like created without necessarily diving into the technical details. 2) Planning - you need to have a high level plan of roughly how you'd like to structure your code. Think of it like you're drawing the outline and AI is filling in the gaps. 3) Separation of concerns - use software principles to drive your code design. Break problems down into separate components, AI is good at filling in components that are well defined. Once you change your thinking to a higher level, then you can maintain flow state. Of course the AI isn't perfect and will make mistakes - you do need to question it as you go. The more creative you become with a solution the harder time the AI will have and sometimes you'll have to break out and fix things up yourself. For me it's a huge productivity boost.
• heisenberg1 2y ago
  This is obvious. Right, of course you get an increase in productivity, especially as a junior - when current AI is able to solve leetcode. BUT I think a lot of people mentioned that, you get code - that the person which wrote it do not understand. So the next time you get a bug there, good luck fixing it. My take so far. AI is great, but only for non critical, non core code. Everything that is done for plotting and scripting is awesome (which can take days to implement and in minutes with AI) - but core lib functions - wouldn't outsource it to the AI right now.
• deleted 2y ago
  [deleted]
• noobermin 2y ago
  Can someone potentially smarter than me explain how the data, which in table I clearly shows the majority of the means for each experiment metric being less than the SD could even hope to be salvaged? Taken blindly, the results are simply unbelieveable to outright lying, the sort of thing you see submitted to garbage open access journals. The text describing model they employ afterwards is not convincing enough for me and seems light on details. I mean, wouldn't any reasonable reviewer demand more? I know preprints don't need polish but this is even below the standard of a preprint, imo.
  • smokel 2y ago
    At least they acknowledge this: "However, the table also shows that for all outcomes (with the exception of the Build Success Rate), the standard deviation exceeds the pre-treatment mean, and sometimes by a lot. This high variability will limit our power in our experimental regressions below." What I find even stranger is that the values in the "control" and "treatment" columns are so similar. That would be highly unlikely given the extreme variability, no?
• smokel 2y ago
  Interestingly, they compare number of pull requests as a statistic for productivity. Not that I know of a better metric, but I wonder if this is an accurate metric. It seems similarly misguided as looking at lines of code. If an AI tool makes me more productive, I would probably either spend the time won browsing the internet, or use it to attempt different approaches to solve the problem at hand. In the latter case, I would perhaps make more reliable or more flexible software. Which would also be almost impossible to measure in a scientific investigation. In my experience, the differences in developer productivity are so enormous (depending on existing domain knowledge, motivation, or management approach), that it seems pretty hard to make any scientific claim based on looking at large groups of developers. For now, I prefer the individual success story.
• ergonaught 2y ago
  The result is "less experienced people got more stuff done". I do not see an assessment of whether the stuff that got done was well done. The output of these tools today is unsafe to use unless you possess the ability to assess its correctness. The less able you are to perform that assessment, the more likely you are to use these tools. Only one of many problems with this direction, but gravity sucks, doesn't it.
• arexxbifs 2y ago
  My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors. My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they've still not (fully) understood. The process of experimenting, learning and understanding is then largely replaced by the LLM, and the real skill becomes applying prompt tweaks until it looks like stuff works.
  • arealaccount 2y ago
    Why I recommend something like CoPilot over ChatGPT to juniors wanting to use AI. At least Copilot you’re still more or less driving, whereas ChatGPT you’re more the passenger and not growing intuition.
  • DanHulton 2y ago
    I was really hoping this study would be exploring that. Really, it's examining short-term productivity gains, ignoring long-term tech debt that can occur, and _completely_ ignoring effects on the growth of the software developers themselves. I share your hunch, though I would go so far as to call it an informed, strong opinion. I think we're going to pay the price in this industry in a few years, where the pipeline of "clueful junior software developers" is gonna dry way up, replaced by a firehose of "AI-reliant junior software developers", and the distance between those two categories is a GULF. (And of course, it has a knock-on effect on the number of clueful intermediate software developers, and clueful senior software developers, etc...)
  • diob 2y ago
    I think it can be abused like anything else (copy paste from stack overflow until it works). But for folks who are going to be successful with or without it, it's a godsend in terms of being able to essentially ask stack overflow questions and get immediate non judgemental answers. Maybe not correct all the time, but that was true with stack overflow as well. So as always, it comes back to the individual.
  • 2y ago
• 29athrowaway 2y ago
  The study doesn't cover the long term effects of using generative AI on a project, which is the deskilling of your workforce. Because development will become an auction-like activity where the one that accepts more suggestions wins.
  • wrink 2y ago
    I think this is right. Just have to jump ship before it all explodes and people wonder what happened Similar to gaming the stock price for a couple quarters as C-level, but now with more incentive for this sort of behavior at IC level
    • 29athrowaway 2y ago
      You cannot jump ship now that every institution has been assimilated by the Borg, aka MBAs. They assimilate companies and leave a bloated hollow mess behind. https://www.youtube.com/watch?v=P4VBqTViEx4 https://www.youtube.com/watch?v=P4VBqTViEx4
  • jillesvangurp 2y ago
    I look at this very differently. There are a lot of grumpy people jealously guarding their "skills" and artisanal coding prowess getting really annoyed at the juniors that come in and devalue what they do by just asking an AI to do the same thing and then moving on with their day. Young people are more mentally agile and most young people are now growing up with LLMs as a tool chain that was always there. I'm actually fairly senior (turning 50 next month) and I notice an effect that AI is having on my own productivity: I now take on mini projects that I used to delegate or avoid doing because they would take too much time or be too tedious. That's not the case anymore. The number of things I can use chat gpt for is gradually expanding. I notice that I'm skilling up a lot more rapidly as well. This is great because if you want to stay relevant, you need to adapt to modern tools and technology. That's nothing new of course. Changes are a constant in our industry. And there always are a lot of people that learn some tricks when they are young and then never learn anything new again. If you are lucky some of that stuff stays relevant for a few decades. But mostly a lot of stuff gets unceremoniously dumped by younger generations. The ability to use LLMs is becoming an important skill in itself and one that is now part of what I look for in candidates (including older ones). I don't have a lot of patience for people refusing to use tools that are available to them. Tell me how you use tools to your advantage; not how you have a tool aversion or are unwilling to learn new things.
• einpoklum 2y ago
  Reminds me of a situation I've been in a few times already: Dev: Hey einpoklum, how do I do XYZ? Me: Hmm, I think I remember that... you could try AB and then C. Dev: Ok, but isn't there a better/easier way? Let me ask ChatGPT. ... Dev: Hey einpoklum, ChatGPT said I should do AB and then C. Me: Let me have a look at that for a second. Me: -Right, so it's just what I read on StackOverflow about this, a couple of years ago. Sometimes it's even the answer that _I_ wrote on StackOverflow and then I feel cheated.
  • slt2021 2y ago
    true, ChatGPT even imports libraries and configs that are 4-5 year outdated.
    • einpoklum 2y ago
      I didn't mean to say the answer is outdated... I try not to give people outdated answers :-\
      • Viliam1234 2y ago
        Neither do I, but those answers later become outdated anyway. :D
  • meiraleal 2y ago
    > Sometimes it's even the answer that _I_ wrote on StackOverflow and then I feel cheated. What's next? Royalties from companies that used the answer in their solutions? Do you also want copyright of this comment?
    • einpoklum 2y ago
      > What's next? Perhaps developing a little sense of humor? :-\
      • meiraleal 2y ago
• zitterbewegung 2y ago
  I am an average developer with more than five years experience in Python. I was using chatgpt to create prototypes of what to do in something I was familiar with and was able to debug the task to make it work. I wasn’t specific enough to specify the epaper display had seven colors instead of black and white. When I was using chatgpt to do qualifiers for a CTF called Hack A Sat at defcon 31 I could not get anything to work such as gnu radio programs. If you have the ability to debug then I have experienced that it is productive but when you don’t understand you run into problems.
• yieldcrv 2y ago
  > Notably, less experienced developers showed higher adoption rates and greater productivity gains. This is what I’ve seen too, I don't think less experienced developers have gotten better in their understanding of anything just more exposed and quicker, while I do think more experience developers have stagnated
  • dpflan 2y ago
    > while I do think more experience developers have stagnated Is this because they are not using coding assistants? Are they resistant to using them? I have to say that the coding assistant is helpful; it is an ever-present rubber duck that can talk back with useful information.
    • yieldcrv 2y ago
      Across teams and organizations what I’ve seen this year is that the “best” developers haven’t looked. The proverbial rockstars with the most domain knowledge have never gotten around to AI and LLMs at all. This is compounded by adherence to misguided corporate policies that broadly prohibit use of LLMs but are meant to only be about putting trade secrets into the cloud, not distinguishing between cloud vs locally run language models. Comfortable people would never challenge this policy with critical thinking, and it requires special interest to look at locally run language models, even just to choose which one to run. Many developers have not advocated for more RAM on their company issued laptop to run better LLMs. and I haven't seen any internal language model that the company is running in their intranet. But it would be cool if there was a huggingface-style catalogue and server farm companies could have and let their employees choose models to prompt, always having the latest models to load.
• simpleranchero 2y ago
  Microsoft people in research team proving Microsoft tools are good. Elsevier will now do ads in research papers.
  • Bjorkbat 2y ago
    This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster. The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average. https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/#figure-summary-of-the-experiment-process-and-results https://github.blog/news-insights/research/research-quantify... And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster. By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.
    • mistrial9 2y ago
      what do these three things have in common? * control the agenda items in a formal meeting * fill a fixed amount of time in an interview with no rebuttal * design the benchmark experiments and the presentation of the results
• ang_cire 2y ago
  I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mean they only needed 1/3 the original roles, and that devs could just also be their operations and appsec team. That's not a knock on shift-left and integrated operations models; those are often good ideas. It's just the logical outcome of those models when execs think they can get a bigger bonus by cutting costs by cutting headcounts. Now you have new devs coming into insanely complex n-microservice environments, being asked to learn the existing codebase, being asked to learn their 5-tool CI/CD pipelines (and that ain't being taught in school), being asked to learn to be DBAs, and also to keep up a steady code release cycle. Is anyone really surprised they are using ChatGPT to keep up? This is going to keep happening until IT companies stop cutting headcounts to make line go up (instead of good business strategy).
  • yifanl 2y ago
    They're cutting headcount because they have no conception of how to make a good product, so reducing costs is the only of making the bottom line go up. That's across the board, from the startups whose business plan is to be acquired at all costs, to the giant tech companies, whose business plan is to get monopoly power first, then figure out how to extract money later.
    • goatlover 2y ago
      Failures of capitalism that probably need regulation. If the middle class and workers are to be protected.
      • 01HNNWZ0MV43FF 2y ago
        Would UBI and LVT do the trick? I'm okay with minimum wages on the idea that a democratic government is the biggest possible de facto labor union, but in general I don't want the government making too many rules about how you can hire and fire people
• godelski 2y ago
  B(i)ased as fuck. I think this post from the other day adds some important context[0]. In that study kids with access to GPT did way more practice problems but worse on the test. But the most important part was that they found that while GPT usually got the final answer right that the logic was wrong, meaning that the answer is wrong. This is true for math and code. There's the joke: there's two types of 10x devs, those that do 10x work and those who finish 10x jira tickets. The problem with this study is the assumptions that it makes, which is quite common and naive in our industry. They assume that PRs and commits are measures of productivity and they assume passing review is a good quality metric. These are so variable between teams. Plenty are just "lgtm" reviews. The issue here is that there's no real solid metric for things like good code. Meeting the goals of a ticket doesn't mean you haven't solved the problem so poorly you are the reason 10 new tickets will be created. This is the real issue here and the only real way to measure it is using Justice Potter's test (I know it when I see it), and requires an expert evaluator. In other words, tech debt. Which is something we're seeing a growing rise in, all the fucking enshitification. So I don't think that study here contradicts [0], in fact I think they're aligned. But I suspect people who are poor programmers (or non programmers) will use this at evidence for what they want to see. Believing naive things like lines of code, number of commits/PRs, etc are measures of productivity rather than hints of measure. I'm all for "move fast and break things" as long as there's time set aside to clean up the fucking mess you left behind. But there never is. It's like we have businesses ADHD. There's so much lost productivity because so much focus is placed on short term measurements and thinking. I know medium and long term thinking are hard, but humans do hard shit every day. We can do a lot better than a shoddy study like this. [0] https://news.ycombinator.com/item?id=41453300 https://news.ycombinator.com/item?id=41453300
• Delk 2y ago
  It's probably worth going a bit deeper into the paper before picking up conclusions. And I think the study could really do a bit of a better job of summarizing its results. The abstract and the conclusion only give a single percentage figure (26.08% increase in productivity, which probably has too many decimals) as the result. If you go a bit further, they give figures of 27 to 39 percent for juniors and 8 to 13 percent for seniors. But if you go deeper, it looks like there's a lot of variation not only by seniority, but also by the company. Beside pull requests, results on the other outcome measures (commits, builds, build success rate) don't seem to be statistically significant at Microsoft, from what I can tell. And the PR increases only seem to be statistically significant for Microsoft, not for Accenture. And even then possibly only for juniors, but I'm not sure I can quite figure out if I've understood that correctly. Of course the abstract and the conclusion have to summarize. But it really looks like the outcomes vary so much depending on the variables that I'm not sure it makes sense to give a single overall number even as a summary. Especially since statistical significance seems a bit hit-and-miss. edit: better readability
  • baxtr 2y ago
    Re 26.08%: I immediately question any study (outside of physics etc) that provides two decimal places.
    • kkyr 2y ago
      Why?
      • jayd16 2y ago
        The assumption is that they almost certainly did not have the sample size to justify two decimal places. If they want two decimals for aesthetics over properly honoring significant figures then it calls the scientific rigor into question.
        • energy123 2y ago
          But they say "26.08% increase (SE: 10.3%)", so they make it clear that there's a lot of uncertainty around that number. They could have said "26% (rounded to 0 dp)" or something, but that conveys even less information about the amount of uncertainty than just saying what the standard error is.
• zone411 2y ago
  This was likely Copilot based on GPT 3.5. Microsoft: September 2022 to May 3rd, 2023 Accenture: July 2023 to December 2023 Anonymous Company: October 2023 to ? Copilot _Chat_ update to GPT-4 was Nov 30, 2023: https://github.blog/changelog/label/copilot/ https://github.blog/changelog/label/copilot/
  • jjcm 2y ago
    Great call out. I'd be extremely curious what the results looked like with something like cursor or just using claude out of the box. I'm amazed at just how easy it is to get simple small scripts up and going now with claude.
• agentultra 2y ago
  Empirical studies like this are hard to conduct... I'm curious though. This study was authored by at least two folks from Microsoft and one of the sample groups in the study was also from Microsoft. The reason this seems to stand out to me as odd is that Microsoft also owns the AI tool being used in the study and would definitely want a favourable conclusion in this paper. Is that a flag we should be watching out for?
  • anonymousDan 2y ago
    It sure is.
  • JonChesterfield 2y ago
    That the paper on gen ai would be marketing of gen ai by the company selling the tool? Yeah, that's a pretty clear conflict of interest.
• deleted 2y ago
  [deleted]
• croes 2y ago
  >Notably, less experienced developers showed higher adoption rates and greater productivity gains. If you start low it's easier to get greater growth rates. The biggest is the first step, 0% to 1% is infinite growth.
• zombiwoof 2y ago
  I sometimes wonder if LeetCode killed the Software Star
• MBCook 2y ago
  It lets people make more PRs. Woohoo. Who cares? Does it increase the number of things that pass QA? Do the things done with AI assistance have fewer bugs caught after QA? Are they easier to extend or modify later? Or do they have rigid and inflexible designs? A tool that can help turn developers into unknown quality code monkeys is not something I’m looking for. I’m looking for a tool that helps developers find bugs or design flaws in what they’re doing. Or maybe write well designed tests. Just counting PRs doesn’t tell me anything useful. But it triggers my gut feeling that more code per unit time = lower average quality.
  • anthomtb 2y ago
    Dev - “Hey copilot, please split this commit into 5 separate commits” Copilot - “okay I can do that for you! Here are your new commits!” Senior Dev - “why? Your change is atomic. I’ll tell management to f-off, in a kind way, if they bring up those silly change-per-month metrics again”
• makk 2y ago
  > Notably, less experienced developers showed higher adoption rates and greater productivity gains. And that is why demand for senior developers is going to go through the roof. Who is going to unfuck the giant balls of mud those inexperienced devs are slinging together? Who’s going to keep the lights on?
• charlie0 2y ago
  On the surface, this appears to be great news. However, there's a big question as to whether these are short productivity gains vs longer lasting gains. There's a hypothesis that the AI generate code will slowly spaghetti-fy a codebase. Is 1-2 years sufficiently long enough to take this into consideration? Or disprove the spaghettification?
• dheerkt 2y ago
  Study funded by Microsoft, conducted by Microsoft engineers, says Microsoft product makes engineers +X% more productive.
• PaulHoule 2y ago
  It beats StackOverflow, but that might be saying more about SO.
• andai 2y ago
  Didn't Microsoft find way higher gains in a previous study? I wonder where the differences (also between companies) come from.
• michaelteter 2y ago
  ChatGPT and to a lesser degree Copilot have been very valuable to me this year. Copilot often saves me a lot of typing on a 1-3 line scope, occasionally surprising me with exactly what I was about to write on a 5-10 line scope. It’s really good during rearrangement and early refactoring (as you are building a new thing and changing your mind as you go about code organization). ChatGPT, or “Jimmy” - as I like to call him - has been great for answering syntax questions, idiom questions, etc. when applying my general skills based on other languages to ones I’m less familiar with. It has also been good for “discussing” architecture approaches to a problem with respect to a particular toolset. With proper guidance and very clear prompting, I usually get highly value responses. I would rough guess that these two tools have saved me 2-3 months of solo time this year - nay, since April. One I get down in the deep details, I use Jimmy much less often. But when I hit something new, or something I long since forgot, he’s ready to be relative expert / knowledge base.
  • iLoveOncall 2y ago
    It saved you 2-3 months over a 5 months period? Unless one is an absolutely terrible developer working on incredibly simple problems, this is simply impossible.
    • michaelteter 2y ago
      I've been doing a lot of new things lately and using some tech I hadn't used before (or lately... and much has changed). Being able to ask for an example of something in that domain, and get a useful answer, is much, much faster than hunting down the current documentation (which may be thin or non-existent). Also being able to say, "I do X in this language. What is the idiomatic way of doing it in Y language?" My pretty broad knowledge can be directed, with careful wording, at ChatGPT, and ChatGPT is the relative domain expert who can get me quite close to a correct solution very quickly.
• clbrmbr 2y ago
  I used to be super productive at the raw keyboard. Then RSI got to me. But with CoPilot, I’m back to my normal productivity. For me it’s a life-saver as it allows fast typing with minimal hand strain.
• throwaway_2494 2y ago
  I think that over time, AI will tend to make all programmers into average programmers. For those who are beginners, it can bring their skills up and make them look like better developers than they are. More insidiously, expert programmers who overuse of AI might also regress to the mean as their skills deteriorate.
• dimal 2y ago
  It matters what you measure. The studies only looked at Copilot usage. I’m an experienced engineer. Copilot is worse than useless for me. I spend most of my time understanding the problem space, understanding the constraints and affordances of the environment I’m in and thinking about the code I’m going to write app. When I start typing code, I know what I’m going to write, and so a “helpful” Copilot autocomplete is just distraction for me. It makes my workflow much much worse. On the other hand, AI is incredibly useful for all of those steps I do before actually coding. And sometimes getting the first draft of something is as simple as a well crafted prompt (informed by all the thinking I’ve done prior to starting. After that, pairing with an LLM to get quick answers for all the little unexpected things that come up is extremely helpful. So, contrary to this report, I think that if experienced developers use AI well, they could benefit MORE than inexperienced developers.
  • danielmarkbruce 2y ago
    Just for clarity - are you saying that going back and forth with ChatGPT is more useful than Co-pilot? The reason I ask is I have both and 95% of the benefit is ChatGPT.
  • dclowd9901 2y ago
    I think my experience mirrors your own. We have access at my job but I’ve turned it off recently as it was becoming too noisy for my focus. I found the tool to be extremely valuable when working in unfamiliar languages, or when doing rote tasks (where it was easy for me to identify if the generated code was up to snuff or not). Where I think it falters for me is when I have a very clear idea of what I want to do, and its _similar_ to a bog standard implementation, but I’m doing something a bit more novel. This tends to happen in “reduce”s or other more nebulous procedures. As I’m a platform engineer though, I’m in a lot of different spaces: Bash, Python, browser, vanilla JS, TS, Node, GitHub actions, Jenkins Java workflows, Docker, and probably a few more. It gives my brain a break while I’m context switching and lets me warm up a bit when I move from area to area.
    • mlinhares 2y ago
      the fact that Copilot hallucinates methods/variables/classes that do not exist in compiled languages where it could know they do not exist is just unbelievable to me. it really feels like people building the product do not care about the UX.
• robodale 2y ago
  I've used Copilot and chatGPT to help with algorithms where I'm unsure where to start. Actual case: "Write a performant algorithm to find the number of work days given a future date". It's trickier than you think and makes a great interview question. Both AI tools came back with...garbage. Loops within loops within loops as they iterated through each day to check if the day is a weekend or not, is a leap year and to account for the extra day, is it a holiday or not, etc. However, chatGPT provided a clever division to cut the dataset down to weeks, then process the result. I ended up using that portion in my final algorithm creation. So, my take on AI coding tools are: "Buyer beware. Your results may vary".
  • zzleeper 2y ago
    I was never able to use them well, but cursor.ai together with Claude is starting to feel actually useful
  • JonChesterfield 2y ago
    How did you cite / reference the clever division it had copied out of the training set?
• matthewtovbin 2y ago
  You can easily evaluate the impact of GitHub Copilot on your org using Faros AI - https://github.com/marketplace/faros-ai https://github.com/marketplace/faros-ai (disclamer - I work there)
• knallfrosch 2y ago
  An interesting topic that will, given this site's community, attract a lot of strong opinions. I, for one, only decide whether CoPilot's productivity increase is worth the $10 it costs per month. It doesn't really matter whether you're an employer getting a 3–30% increase in productivity or whether you pay for it personally and finish 2 hours faster every week and log off illegaly. It's easily worth its money. What more to consider?
  • ThalesX 2y ago
    > What more to consider? Potentially sharing company IP with a 3rd party?
• throwaway918299 2y ago
  The effect it’s had on my company: firing all of the over employed retail devs because all they submit is GPT slop that doesn’t even work.
• JonChesterfield 2y ago
  It's very exciting that generative ai lets people get more code written, especially in unfamiliar domains. It feels great to ship loads of code that does a thing. The sure result is an unprecedented increase in the amount of code in products. A minor drawback to that enthusiasm is that a lot of the code I read didn't need to exist in the first place, even before this wave. Lots of it can be attributed to the path dependence of creation as opposed to what it is trying to do. This should be a rich time to change to security / exploit work - the random search tools are great and the target just keeps getting easier. What our industry really desperately needed was to drive the quality of implementation right down. It's going to be an exciting time to be alive.
• aussieguy1234 2y ago
  "Numbers of completed tasks" might not be the best metric to use here. No two tasks are the same level of complexity and one task may take 5x longer than another to complete.
• joeyguerra 2y ago
  Probably similar results as pair programming.
• Swiftdream 2y ago
  [flagged]