14 ms·

AI coding tools can reduce productivity

279 points by gk1 1y ago

latenightcoding 1y ago
LLMs make me 10-20x more productive in frontend work which I barely do. But when it comes to low-level stuff (C/C++) I personally don't find it too useful. it just replaces my need to search stackoverflow. edit: should have mentioned the low-level stuff I work on is mature code and a lot of times novel.
- justinko 1y ago
  Same. It’s amazing for frontend.
  - relaxing 1y ago
    Is this because they had the entire web to train on, code + output and semantics in every page?
    - Falimonda 1y ago
      It's moreso that a backend developer can now throw together a frontend and vice-versa without relying on a team member or needing to set aside time to internalize all the necessary concepts to just make that other part of the system work. I imagine even a full-stack developer will find benefits.
      - hluska 1y ago
        This has nothing to do with what they asked.
        Falimonda 1y ago
        Copilot is going to feel "amazing" at helping you quickly work within just about any subject that you're not already an expert in. Whether or not a general purpose foundation model for coding is trained on more backend or frontend code is largely irrelevant in this specific context.
        relaxing 1y ago
        Found the bot.
      - owebmaster 1y ago
        So we are all back to be webmasters :)
    - hluska 1y ago
      I’m not sure how this was extended and refined but there are sure a lot of signs of open source code being used heavily (at least early on). It would make sense to test model fit with the web at large.
    - xigoi 1y ago
      I guess it’s because modern front-end “development” is mostly about copying huge amounts of pointless boilerplate and slightly modifying it, which LLMs are really good at.
  - famahar 1y ago
    It's astonishing. A bit scary actually. Can easily see the role of front-end slowly morphing into a single person team managing a set of AI tools. More of an architecture role.
    - justinko 1y ago
      A yes I’m more of a backend guy that loves tailwind.
  - Brendinooo 1y ago
    As a front-of-the-frontend guy, I think it's terrible with CSS and SVG and just okay with HTML. I work at a shop where we do all custom frontend work and it's just not up to the task. And, while it has chipped in on some accessibility features for me, I wouldn't trust it to do that unsupervised. Even semantic HTML is a mixed bag: if you point out something is a figure/figcaption it'll probably do it right, but I haven't found that it'll intuit these things and get it right on the first try. But I'd imagine if you don't care about the frontend looking original or even good, and you stick really closely to something like tailwind, it could output something good enough. And critically, I think a lot of times the hardest part of frontend work is starting, getting that first iteration out. LLMs are good for that. Actually got me over the hump on a little personal page I made a month or so ago and it was a massive help. Put out something that looked terrible but gave me what I needed to move forward.
- sottol 1y ago
  Interesting, I find the exact opposite. Although to a much lesser extent (maybe 50% boost). I ended shoehorned into backend dev in Ruby/Py/Java and don't find it improves my day to day a lot. Specifically in C, it can bang out complicated but mostly common data-structures without fault where I would surely do one-off errors. I guess since I do C for hobby I tend to solve more interesting and complicated problems like generating a whole array of dynamic C-dispatchers from a UI-library spec in JSON that allows parsing and rendering a UI specified in YAML. Gemini pro even spat out a YAML-dialect parser after a few attempts/fixes. Maybe it's a function of familiarity and problems you end using the AI for.
  - freeone3000 1y ago
    As in, it seems to be best at problems that you’re unfamiliar with in domains where you have trouble judging the quality?
    - Brendinooo 1y ago
      >it seems to be best at problems that you’re unfamiliar with Yes. >in domains where you have trouble judging the quality Sure, possibly. Kind of like how you think the news is accurate until you read a story that's in your field. But not necessarily. Might just be more "I don't know how do to <basic task> in <domain that I don't spend a lot of time in>", and LLMs are good at doing basic tasks.
- moron4hire 1y ago
  This feels like a parallel to the Gell-Mann amnesia effect. Recently, my company has been investigating AI tools for coding. I know this sounds very late to the game, but we're a DoD consultancy and one not traditional associated with software development. So, for most of the people in the company, they are very impressed with the AI's output. I, on the other hand, am a fairly recent addition to the company. I was specifically hired to be a "wildcard" in their usual operations. Which is too say, maybe 10 of us in a company of 3000 know what we're doing regarding software (but that's being generous because I don't really have visibility into half of the company). So, that means 99.7% of the company doesn't have the experience necessary to tell what good software development looks like. The stuff the people using the AI are putting out is... better than what the MilOps analysts pressed into writing Python-scripts-with-delusions-of-grandeur were doing before, but by no means what I'd call quality software. I have pretty deep experience in both back end and front end. It's a step above "code written by smart people completely inexperienced in writing software that has to be maintained over a lifetime", but many steps below, "software that can successfully be maintained over a lifetime".
  - IX-103 1y ago
    Well, that's what you'd expect from an LLM. They're not designed to give you the best solution. They're designed to give you the most likely solution. Which means that the results would be expected to be average, as "above average" solutions are unlikely by definition. You can tweak the prompt a bit to skew the probability distribution with careful prompting (LLMs that are told to claim to be math PHDs are better at math problems, for instance), but in the end all of those weights in the model are spent to encode the most probable outputs. So, it will be interesting to see how this plays out. If the average person using AI is able to produce above average code, then we could end up in a virtuous cycle where AI continuously improves with human help. On the other hand, if this just allows more low quality code to be written then the opposite happens and AI becomes more and more useless.
    - leptons 1y ago
      I have no doubt which way it is going to go.
- kannanvijayan 1y ago
  I've been hacking on some somewhat systemsy rust code, and I've used LLMs from a while back (early co-pilot about a year ago) on a bunch of C++ systems code. In both of these cases, I found that just the smart auto-complete is a massive time-saver. In fact, it's more valuable to me than the interactive or agentic features. Here's a snippet of some code that's in one of my recent buffers: // The instruction should be skipped if all of its named // outputs have been coalesced away. if ! self.should_keep_instr(instr) { return; } // Non-dropped should have a choice. let instr_choice = choices.maybe_instr_choice(instr_ref) .expect("No choice for instruction"); self.pick_map.set_instr_choice( instr_ref, instr_choice.clone(), ); // Incref all named def inputs to the PIR choice. instr_choice.visit_input_defs(|input_def| { self.def_incref(input_def); }); // Decref all named def inputs to the SIR instr. instr.visit_inputs( |input_def| self.def_decref(input_def, sir_graph) ); The actual code _I_ wrote were the comments. The savings in not having to type out the syntax is pretty big. About 80% of the time in manual coding would have been that. Little typos, little adjustments to get the formatting right. The other nice benefit is that I don't have to trust the LLM. I can evaluate each snippet right there and typically the machine does a good job of picking out syntactic style and semantics from the rest of the codebase and file and applying it to the completion. The snippet, if it's not obvious, is from a bit of compiler backend code I'm working on. I would never have even _attempted_ to write a compiler backend in my spare time without this assistance. For experienced devs, autocomplete is good enough for massive efficiency gains in dev speed. I still haven't warmed to the agentic interfaces because I inherently don't trust the LLMs to produce correct code reliably, so I always end up reviewing it, and reviewing greenfield code is often more work than just writing it (esp now that autocomplete is so much more useful at making that writing faster).
  - sgc 1y ago
    What exact tool are you using for your smart auto-complete?
    - kannanvijayan 1y ago
      Whatever copilot defaults to doing on vscode these days. I didn't configure it very much - just did the common path setup to get it working.
- cguess 1y ago
  This is exactly my experience as well. I've had agents write a bit of backend code, always small parts. I'm lucky enough to be experienced enough with code I didn't write to be able to quickly debug it when it fails (and it always fails from the first run). Like using AI to write a report, it's good for outlines, but the details are always seemingly random as far as quality. For frontend though? The stuff I really don't specialize in (despite some of my first html beginning on FrontPage 1997 back in 1997), it's a lifesaver. Just gotta be careful with prompts since so many front end frameworks are basically backend code at this point.
- AstroBen 1y ago
  This is good if front end is something you just need to get through. It's terrible if your work is moving to involve a lot of frontend - you'll never pick up the skills yourself
- sysmax 1y ago
  It works with low-level C/C++ just fine as long as you rigorously include all relevant definitions in the context window, provide non-obvious context (like the lifecycle of some various objects) and keep your prompts focused. Things like "apply this known algorithm to that project-specific data structure" work really well and save plenty of time. Things that require a gut feeling for how things are organized in memory don't work unless you are willing to babysit the model.
- sensanaty 1y ago
  As the fullstacker with a roughly 65/35 split BE/FE on the team who has to review this kinda stuff on the daily, there's nothing I dread more than a backender writing FE tickets and vice versa. Just last week I had to review some monstrosity of a FE ticket written by one of our backenders, with the comment of "it's 90% there, should be good to takeover". I had to throw out pretty much everything and rewrite it from scratch. My solution was like 150 lines modified, whereas the monstrous output of the AI was non-functional, ugly, a performance nightmare and around 800 lines, with extremely unhelpful and generic commit messages to the tune of "Made things great!!1!1!!". I can't even really blame them, the C-level craze and zeal for the AI shit is such that if you're not doing crap like this you get scrutinized and PIP'd. At least frontenders usually have some humility and will tell you they have no clue if it's a good solution or not, while BEnders are always for some reason extremely dismissive of FE work (as can be seen in this very thread). It's truly baffling to me
freetime2 1y ago
Here is the the methodology of the study: > To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study. So it's a small sample size of 16 developers. And it sounds like different tasks were (randomly) assigned to the no-AI and with-AI groups - so the control group doesn't have the same tasks as the experimental group. I think this could lead to some pretty noisy data. Interestingly - small sample size isn't in the list of objections that the auther includes under "Addressing Every Objection You Thought Of, And Some You Didn’t". I do think it's an interesting study. But would want to see if the results could be reproduced before reading into it too much.
- Tainnor 1y ago
  The sample size isn't 16 developers, it's 246 issues.
  - freetime2 1y ago
    So agree with that - but on the other hand surely the number of developers matters here? For example, if instead of 16 developers the study consisted of a single developer completing all 246 tasks with or without AI, and comparing the observed times to complete, I think most people would question the reproducibility and relevancy of the study?
    - Tainnor 1y ago
      It matters in the sense that it is unclear whether the findings generalise to other people. Which is a problem that a lot of studies, even with more participants, have because they may not have a diverse enough set of participants. But in terms of pure statistical validity, I don't think it matters.
budududuroiu 1y ago
I was surprised at how much better v0 was these days. I remember it yielding clunky UIs initially. I thought it was the model, but then I realised, v0 is carried by the shadcn UI library, not the intelligence of the model
JKCalhoun 1y ago
As others probably have experienced, I can only add that I am doing coding now I would have kicked down the road if I did not have LLM assistance. Example: using LeafletJS — not hard, but I didn't want to have to search all over to figure out how to use it. Example: other web page development requiring dropping image files, complicated scrolling, split-views, etc. In short, there are projects I have put off in the past but eagerly begin now that LLMs are there to guide me. It's difficult to compare times and productivity in cases like that.
- georgemcbay 1y ago
  This is pretty similar to my own experience using LLMs as a tool. When I'm working with platforms/languages/frameworks I am already deeply familiar with I don't think they save me much time at all. When I've tried to use them in this context they seem to save me a bunch of time in some situations, but also cost me a bunch of time in others resulting in basically a wash as far as time saved goes. And for me a wash isn't worth the long-term cost of losing touch with the code by not being the one to have crafted it. But when it comes to environments I'm not intimately familiar with they can provide a very easy on-ramp that is a much more pleasant experience than trying to figure things out through often iffy technical documentation or code samples.
- timeon 1y ago
  > search all over to figure out how to use it. Leaflet doc is single page document with examples you can copy-paste. There is page navogation at the top. Also ctrl/cmd+f and keyword seems quicker than writing the prompt.
  - JKCalhoun 1y ago
    Nice. I'm afraid I simply assumed, like other "frameworks", it was going to entail wandering all over StackOverflow, etc. Still, when I simply told Claude that I wanted the pins to group together when zoomed out — it immediately knew I meant "clustering" and added the proper import to the top of the HTML file ... got it done.
tomcam 1y ago
What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.
- journal 1y ago
  what about the $ you make? isn't that an indicator? you've probably made more than me, so you are more successful while both of us might be doing the same thing.
  - __MatrixMan__ 1y ago
    I don't think there's much of a correlation there.
    - __MatrixMan__ 1y ago
      A better measure would be how much work has been made unnecessary. The asymptote approached by software engineering is GDP=$0 because all problems are solved by maximally efficient automation. Never gonna happen, but progress on that path is a decent proxy for efficiency. Too often the job is about introducing problems that are good for some company's bottom line, but that's the opposite of efficiency.
  - hammyhavoc 1y ago
    Productivity has zero to do with salary. Case in point: FOSS. Some of the most productive devs don't get paid by the big corps who make use of their open source projects, hence the constant urging of corps and people to sponsor projects they make money via.
  - Ancalagon 1y ago
    In a vacuum I don’t believe pay alone is a very good indicator. What might be a better one is if someone has a history across their career of delivering working products to spec, doing this across companies and with increasing responsibility. This of course can only be determined after the fact.
hluska 1y ago
I’ve been around tech for a long time. At this point, I’ve lost count of how many hype cycles I’ve seen hit the “hold on, everything sucks” stage. Generative AI is seemingly at the hold on, everything sucks stage and it’s getting repetitive.
- CaptainFever 1y ago
  Trough of Disillusionment (followed by the Slope of Enlightenment and Plateau of Productivity): https://en.wikipedia.org/wiki/Gartner_hype_cycle https://en.wikipedia.org/wiki/Gartner_hype_cycle
  - bluefirebrand 1y ago
    My bold prediction is that the Trough of Disillusionment for LLMs is going to be a very long stretch
- deleted 1y ago
  [deleted]
softwaredoug 1y ago
What if this is true? And then we as a developer community are focused on the wrong thing to increase productivity? Like what if by focusing on LLMs for productivity we just reinforce old-bad habits, and get into a local maxima... And even worse, what if being stuck with current so-so patterns, languages, etc means we don't innovate in language design, tooling, or other areas that might actually be productivity wins?
- journal 1y ago
  imagine having interstate highways built in one night you wake up and you have all these highways and roads and everyone is confused what they are and how to use them. using llm is the opposite of boiling frogs because you're not the leader writing, you're just suggesting... i just realized i might not know what im talking about.
- __MatrixMan__ 1y ago
  We were stuck near local maxima since before LLM's came on the scene. I figure the same concentration of innovators are gonna innovate, now LLM assisted, and the same concentration of best-practice folk are gonna best-practice--now LLM assisted. Local maxima might get sticker, but greener pastures will be found more quickly than ever. I expect it'll balance.
  - softwaredoug 1y ago
    Honestly the biggest hindrance of developer productivity right now is probably perpetual, looming layoffs, not lacking AI, tools, programming languages, etc :)
raggi 1y ago
They averaged producing 47% more code on the AI tasks, but took only 20% more time. The report here biases over these considerations, but I’m left wondering: was the extra code superfluous or did this produce better structure / managed debt better? If that extra 47% of code translates to lower debt and more consistent throughput over the long term, I might take it, given how crushed projects get from debt. Anyway, it’s all hyperbole because there are massive statistical differences in the outcomes but no measures as to what they mean, but I’m sure they have meaning. That meaning matters a ton.
- gpm 1y ago
  Honestly my experience from using AI to code (primarily claude sonnet) is that that "extra 47%" is probably itself mostly tech debt. Places where the AI repeated itself instead of using a loop. Places where the AI wrote tests that don't actually test anything. Places where the AI failed to produce a simple abstraction and instead just kept doing the same thing by hand. Etc. AI isn't very good at being concise, in my experience. To the point of producing worse code. Which is a strange change from humans who might just have a habit of being too concise, but not by the same degree.
  - raggi 1y ago
    Your response implies the ai produced code was landed without review. That’s a possible outcome but I would hope it’s unlikely to account for the whole group at this scale. We’re of course still lacking data.
    - trollbridge 1y ago
      I very much doubt that when individual programmers are producing significantly more code with the help of AI that somehow the review process simultaneously scales up to perform adequate review of all of that extra code. In my experience, review was inadequate back before we had AI spewing forth code of dubious quality. There's no reason to think it's any better now. An actually-useful AI would be one that would make reviews better, do them itself, or at least help me get through reviews faster.
kylecazar 1y ago
Now do a study that specifically gauges how useful an LLM (including smart tab completion) is for a frontend dev working in react/next/tailwind on everyday Jira tickets. These were maintainers of large open source projects. It's all relative. It's clearly providing massive gains for some and not as much for others. It should follow that it's benefit to you depends on who you are and what you are working on. It isn't black and white.
- cheeze 1y ago
  As a backend dev who owns a few internal crappy frontends, LLMs have been the best thing ever. Code quality isn't the top priority, I just need to plumb some data to an internal page at BigCorp.
  - distalx 1y ago
    Could you share more about your process and how they specifically help you with your internal frontends? Any details would be great! Thanks!
- franciscop 1y ago
  It's a very well controlled study about... what the study claims to do. Yes, they didn't study a different thing, for _many_ reasons. Yes, we shouldn't haphazardly extrapolate to other parts of Engineering. But it looks like it's a good study nonetheless. There are some very good findings though, like how the devs thought they were sped up but they were actually slowed down.
  - kylecazar 1y ago
    Agreed. My comment was anticipating the extrapolation part :)
- timeon 1y ago
  React and tailwind already made lot of tradeoffs to make it more ergonomic for developers. One would expect that LLMs could unlock lean and faster stack instead.
xarope 1y ago
I think this for me is the most worrying: "You can see that for AI Allowed tasks, developers spent less time researching and writing code". My analogy to this is seeing people spend time trying to figure out how to change colors, draw shapes in powerpoint, rather than focus on the content and presentation. So here, we have developers now focusing their efforts on correcting the AI output, rather than doing the research and improving their ability to deliver code in the future. Hmm...
- hammyhavoc 1y ago
  This has been my observation too. It's a tool for the lazy.
  - yukai 1y ago
    laziness is a driving force of progress
    - anon15123 1y ago
      in what direction
      - baq 1y ago
        All of them.
        badsectoracula 1y ago
        So the net result is we remain static? :-P
        baq 1y ago
        See also: WALL-E
  - jack_pp 1y ago
    You can say the same about a printer. Or a kindle, oh you're too lazy to carry around 5 books with you?
dearilos 1y ago
I found that early and often code reviews can offset the reduction in productivity. A good code review process can fix this.
strangescript 1y ago
This entire concept hinges on AI not getting better. If you believe AI is going continue to get better at the current ~5-10% a month range, then hand waiving over developer productivity today is about the same thing as writing an article about the internet being a fad in 1999.
- trashchomper 1y ago
  On the flip side, why would I use AI today if it presents no immediate benefit. Why not wait 5 years and see if it becomes actually helpful.
  - strangescript 1y ago
    better yet, wait 10, let me know how it goes
- benrutter 1y ago
  If they do improve at 5-10% a month then that'd definitely be true (tbh I'm not sure they are even improving at that rate now - 10% for a year would be 3x improvement with compounding). I guess the tricky bit is, nobody knows what the future looks like. "The internet is a fad" in 1999 hasn't aged well, but a lot of people touted 1960s AI, XML and 3d telivisions as things that'd be the tools in only a few years. We're all just guessing till then.
dismalaf 1y ago
I find LLMs are decent at regurgitating boilerplate. Basically the same kind of stuff you could google then copy-paste... AI chatbots, now that they have web access, are also good at going over documentation and save you a little time searching through the docs yourself. They're not great at business logic though, especially if you're doing anything remotely novel. Which is the difficult part of programming anyway. But yeah, to the average corporate programmer who needs to recreate the same internal business tool that every other company has anyway, it probably saves a lot of time.
- trollbridge 1y ago
  They're great at helping me figure out how to make something work with a poorly-documented, buggy framework, which is indeed a large fraction of my job, whether I like it or not.
- handfuloflight 1y ago
  This isn't true, and I know it by what I'm working on and sorry, I'm not at liberty to give more details. But I see how untrue this is, every working hour of every day.
  - dismalaf 1y ago
    You say more details as if you gave any to begin with...
    - handfuloflight 1y ago
      Here's a hint: What I input for inference is not in the training data. But the model can generalize well enough to handle the task.
      - dismalaf 1y ago
        > What I input for inference is not in the training data ??? Like everyone else? This is why LLMs have large context windows. > But the model can generalize well enough to handle the task. Sounds like what you're doing is normal corporate business logic. If it's able to generalize it it's probably not particularly novel.
calrain 1y ago
I've been using Claude Code heavily for about 3 months now, and I'm pretty sure I'm between 10 and 20 times more productive while using it. How I measure performance is how many features I can implement in a given period of time. It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.
- fuomag9 1y ago
  Same, I’ve done stuff that should have taken me 2-3 weeks in days
  - fourthark 1y ago
    I’ve done this without AI. The thing was not as hard as I thought it would be.
    - tracker1 1y ago
      Same, often a missing feature or library had become available since I last looked into a subject.
- zsoltkacsandi 1y ago
  I have exactly the same experience.
- benreesman 1y ago
  I find the swings to be wild, when you win with it, you win really big. But when you lose with it, it's a real bite out of your week too. And I think 10x to 20x has to be figurative right, you can do 20x by volume maybe, but to borrow an expression from Steve Ballmer, that's like measuring an airplane by kilograms. Someone already operating at the very limit of their abilities doing stuff that is for them high complexity, high cognitive load, detail intense, and tactically non-obvious? Even a machine that just handed you the perfect code can't 20x your real output, even if it gave you the source file at 20x your native sophistication you wouldn't be able to build and deploy it, let alone make changes to it. But even if it's the last 5-20% after you're already operating at your very limit and trying to hit your limit every single day is massive, it makes a bunch of stuff on the bubble go from "not realistic" to "we did that".
_jhk 1y ago
[dead]
camgunz 1y ago
I finally took the plunge and did a big chunk of work in Cursor. It was pretty ideal: greenfield but with a very relevant example to slightly modify (the example pulled events over HTTP as a server and I wanted it to pull events over Google pub/sub instead). Over IDK, 2-3 hours I got something that seemed on its face to work, but: - it didn't use the pub/sub API correctly - the 1 low-coverage test it generated didn't even compile (Go) - there were a bunch of small errors it got confused by--particularly around closures I got it to "90%" (again though it didn't at all work) with the first prompt, and then over something like a dozen more mostly got it to fix its own errors. But: - I didn't know the pub/sub API--I was relying on Cursor to do this correctly--and it totally submarined me - I had to do all the digging to get the test to compile - I had to go line by line and tell it to rewrite... almost everything I quit when I realized I was spending more time prompting it to fix things than it would take me to fully engage my brain and fix them myself. I also noticed that there was a strong pull to "just do one more prompt" rather than dig in and actually understand things. That's super problematic to me. Worse, this wasn't actually faster. How do I know that? The next day I did what I normally do: read docs and wrote it myself. I spent less time (I'm a fast typist and a Vim user) overall, and my code works. My experience matches pretty well w/ the results of TFA. --- Something I will say though is there is a lot of garbage stuff in tech. Like, I don't want to learn Terraform (again) just to figure out how to deploy things to production w/o paying a Heroku-like premium. Maybe I don't want to look up recursive CTEs again, or C function pointers, or spent 2 weeks researching a heisenbug I put into code for some silly reason AI would have caught immediately. I am _confident_ we can solve these things without boiling oceans to get AI to do it for us. But all this shit about how "I'm 20x more productive" is totally absurd. The only evidence we have of this is people just saying it. I don't think a 20x productivity increase is even imaginable. Overall productivity since 1950 is up 3.6x [0]. These people are asking us to believe they've achieved over 400 years of productivity gains in "3 months". Extraordinary claims require extraordinary evidence. My guess is either you were extremely unproductive before, or (like others are saying in the threads) in very small ways you're 20x more productive but most things are unaffected or even slower. [0]: https://fred.stlouisfed.org/series/OPHNFB https://fred.stlouisfed.org/series/OPHNFB
- bonsai_bar 1y ago
Amaury-El 1y ago
The more I used it, the easier it became to skip over things I should have thought through myself. But looking back, the results weren’t always faster or better. Now I prefer to treat AI as a kind of challenger. It helps reveal the parts I haven't truly understood, rather than just speeding things up.
vouaobrasil 1y ago
AI could make me more productive, I know that for a fact. But, I don't want to be more productive because the tasks that could be automated with AI are those I find enjoyable. Not always in an intellectual sense, but in a meditative sense. And if I automated those away, I think I would become less human.
Fraterkes 1y ago
I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask. That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless. Conversely there's people who love ai and use it for everything, and since the majority of the stuff they ask about is fairly simple and well documented (eg "Write me some typescript"), they rarely have a negative experience.
- aleph_minus_one 1y ago
  I think there are also other aspects: - Some people simply ask a lot more questions than others (this ignores whether they like or dislike AI), i.e. some people simply prefer to find things out more by themselves, and thus also use other resources like Google or Stack Overflow as a last resort. So their questions to an AI will likely be more complicated, because they already found out the easy parts by themselves. - If I have to make the effort to explain to the AI in a sufficiently exhaustive way what I need (which I often have to do), I expect the answers of the AI to be really good. If it isn't, having explained my problem to the AI was simply a waste of time.
  - deleted 1y ago
    [deleted]
  - Tainnor 1y ago
    > I expect the answers of the AI to be really good. If it isn't, having explained my problem to the AI was simply a waste of time. I find the worst part to be when it doesn't correct flaws in my assumptions. For example, yesterday I asked it "what is the difference between these two Datadog queries"? And it replied something that was semi-correct, but it didn't discover the fundamental flaw - that the first one wasn't a valid query because of unbalanced parens. In fact, it turns out that the two strings (+ another one) would get concatenated and only then would it be a valid query. A simple "the first string is not a valid query because of a missing closing paren" would have saved a lot of time in trying to understand this, and I suspect that's what I would have received if I had prompted it with "what's the problem with this query" but LLMs are just too sycophantic to help with these things.
sheepfacts 1y ago
Perhaps is difficult to measure personal productivity in programming, but we can measure that we will run more slowly with 10 kg. in our backpack. I propose this procedure: The SWE selects 10 tasks and guesses some measure of their complexity (time to finish them) and then he randomly select 5 to be done with AI and the rest without. He performs them and finally calculates a deviation D. The deviation D = D_0 - D_1 where D_i = sum (real_time/guessed_time - 1), where D_0 is using AI and D_1 is without AI, the sign and magnitude of D measure respectively if the use of AI is beneficial or detrimental and the impact of using AI. Also, clipping individuals addends to be in the interval [-0.5,0.5] should avoid one bad guess dominating the estimation. Sorry if this is a trivial ideal but it is feasible and intuitively should provide useful information if the tasks are taken among the ones in which each initial guessing has small deviation. A filter should be applied to tasks in which scaffolding by AI surpass a certain relative threshold in case we are interested in generalizing our results to tasks in which scaffolding is not dominating time. It could happen that the impact of using AI depends of the task at hand, the capability of the SWE to pair programming with it, and of the LLM used, to such an extend that those factors were bigger that the average effect on a bag of tasks, in this case the large deviation from the mean makes any one parameter estimation void of useful information.
- didibus 1y ago
  That's pretty much what the study the article refers too did, and it found the use of AI was 19% slower.
titaniumrain 1y ago
history repeats itself - "horses are more efficient than cars" In addition, a study based on 16 devs is representative enough to draw this conclusion?
_heimdall 1y ago
I have never found a measure of programmer productivity that makes sense to me, but I can say that LLM coding tools are way more distracting to me than they are worth. They constantly guess at what I may type next, are often wrong, and pop in with suggestions breaking my mental flow and making me switch from the mindset of coding to the mindset of reviewing code.
Brendinooo 1y ago
The article brushed aside devs being terrible at estimates, but I dunno. I'm a frontend guy, been using Claude Code for a couple of weeks now. It's been able to speed up some boilerplate, it's sped up a lot of "naming is hard" conversations I like to have (but my coworkers probably don't, lol), it's enabled me to do a lot more stuff in my most recent project. But for a task or two I suspect that it has slowed me down. If I'm unable to articulate the problem well enough and the problem is hard enough you can go in circles for awhile. And I think the nature of "the right answer is just around the corner" makes it hard to timebox or find a specific point where you say "yup, time to ditch this and do it the old-fashioned way". There is a bit of a slot-machine effect here.
- Jensson 1y ago
  > But for a task or two I suspect that it has slowed me down Likely more, as it takes longer for you to activate your brain when your first thought is to ask an LLM rather than solve it yourself. Its like people reaching for a calculator to do 4+5, that doesn't make you faster or more accurate.
  - Brendinooo 1y ago
    Depends on if the problem is 4+5 or something much more complex.
Fastjur 1y ago
> To compute the actual speedup – or, rather, slowdown! – provided by AI tools, the researchers compared the developers’ predictions of how long each task would take to the measured completion time. I'm sorry, but it feels to me like this research has only proven that developers tend to underestimate how long a task is supposed to take, with or without AI. In no way did they actually measure how much faster a specific task was when performed with and without AI?
- didibus 1y ago
  What I understand they did is. You have two tasks: - Task 1 - 3h - Task 2 - 1h You ask the dev to estimate both. Then you randomly tell the dev, ok do Task 1 without AI, and Task 2 with AI. Then you measure the actual time it took. Their estimate for AI task missed the mark by 19%, but those without AI were done 20% faster then estimated. At the time of estimating they didn't know if the task would need to be done with AI or not.
  - Fastjur 1y ago
    Right, that makes more sense indeed.
anupj 1y ago
[flagged]
b0a04gl 1y ago
[dead]
greatgib 1y ago
Really a great piece of work. At the opposite of the usual studies posted here. With the headline we can easily guess that the study should be flawed, with the sample not representative, or developers not expert enough with the AI or so, and then, they give a very well done list of all valable objections with arguments about why they don't think that contradict the study. That replied to all the questions I could have had in the end.
SkyPuncher 1y ago
One thing I find frustrating with these conversations is the _strict_ focus on single-task productivity. Arguably, on a single coding task, I don't really move that much faster. However, I have much, much more brain capacity left both while coding and when I'm done coding. This has two knock on effects: 1. Most simply, I'm productive for longer. Since LLMs are doing a lot of the heavy lifting, my brain doesn't have to constantly think. This is especially important in time periods where I'd previously have too little mental energy to think deeply about code. 2. I can do other things while coding. Well, right now, Cursor is cycling on a simple task while I type this. Most days, though, I'm responding to customers, working on documentation/planning, or doing some other non-coding task that's critical to my workflows. This is actually where I find my biggest productivity gains. Instead of coding THEN X, I can now do coding WITH X.
- bluefirebrand 1y ago
  > I can do other things while coding Context shifting while trying to code seems like a bad idea to me Maybe you're some incredible multi -tasking genius able to change tasks rapidly without losing any of the details, but I suspect if most people tried this workflow they would produce worse code and also whatever their other output is would be low quality too
  - SkyPuncher 1y ago
    For me, the most time consuming parts of coding don’t require deep meaningful thought. It’s often things like stepping through code to figure out where something breaks (answer is often obvious once you find it) or iterating through broken specs or simply playing with a library/interface enough to figure out how it actually works (vs how’s it’s documented) This type of stuff is great to have an LLM tackle while I work on something else.
bhaktatejas922 1y ago
I have never felt so disconnected from the findings of a study
bhaktatejas922 1y ago
A lot of this stems from friction in brittle process around applying edits or latency from full file rewrites. The bar to beat is time it takes for human intelligence + Cursor Tab to complete a task, but in practice the bar becomes "feeling" like its faster

279 points by gk1 1y ago

LLMs make me 10-20x more productive in frontend work which I barely do. But when it comes to low-level stuff (C/C++) I personally don't find it too useful. it just replaces my need to search stackoverflow. edit: should have mentioned the low-level stuff I work on is mature code and a lot of times novel.

justinko 1y ago
Same. It’s amazing for frontend.
- relaxing 1y ago
  Is this because they had the entire web to train on, code + output and semantics in every page?
  - Falimonda 1y ago
    It's moreso that a backend developer can now throw together a frontend and vice-versa without relying on a team member or needing to set aside time to internalize all the necessary concepts to just make that other part of the system work. I imagine even a full-stack developer will find benefits.
    - hluska 1y ago
      This has nothing to do with what they asked.
      - Falimonda 1y ago
        Copilot is going to feel "amazing" at helping you quickly work within just about any subject that you're not already an expert in. Whether or not a general purpose foundation model for coding is trained on more backend or frontend code is largely irrelevant in this specific context.
        relaxing 1y ago
        Found the bot.
    - owebmaster 1y ago
      So we are all back to be webmasters :)
  - hluska 1y ago
    I’m not sure how this was extended and refined but there are sure a lot of signs of open source code being used heavily (at least early on). It would make sense to test model fit with the web at large.
  - xigoi 1y ago
    I guess it’s because modern front-end “development” is mostly about copying huge amounts of pointless boilerplate and slightly modifying it, which LLMs are really good at.
- famahar 1y ago
  It's astonishing. A bit scary actually. Can easily see the role of front-end slowly morphing into a single person team managing a set of AI tools. More of an architecture role.
  - justinko 1y ago
    A yes I’m more of a backend guy that loves tailwind.
- Brendinooo 1y ago
  As a front-of-the-frontend guy, I think it's terrible with CSS and SVG and just okay with HTML. I work at a shop where we do all custom frontend work and it's just not up to the task. And, while it has chipped in on some accessibility features for me, I wouldn't trust it to do that unsupervised. Even semantic HTML is a mixed bag: if you point out something is a figure/figcaption it'll probably do it right, but I haven't found that it'll intuit these things and get it right on the first try. But I'd imagine if you don't care about the frontend looking original or even good, and you stick really closely to something like tailwind, it could output something good enough. And critically, I think a lot of times the hardest part of frontend work is starting, getting that first iteration out. LLMs are good for that. Actually got me over the hump on a little personal page I made a month or so ago and it was a massive help. Put out something that looked terrible but gave me what I needed to move forward.
sottol 1y ago
Interesting, I find the exact opposite. Although to a much lesser extent (maybe 50% boost). I ended shoehorned into backend dev in Ruby/Py/Java and don't find it improves my day to day a lot. Specifically in C, it can bang out complicated but mostly common data-structures without fault where I would surely do one-off errors. I guess since I do C for hobby I tend to solve more interesting and complicated problems like generating a whole array of dynamic C-dispatchers from a UI-library spec in JSON that allows parsing and rendering a UI specified in YAML. Gemini pro even spat out a YAML-dialect parser after a few attempts/fixes. Maybe it's a function of familiarity and problems you end using the AI for.
- freeone3000 1y ago
  As in, it seems to be best at problems that you’re unfamiliar with in domains where you have trouble judging the quality?
  - Brendinooo 1y ago
    >it seems to be best at problems that you’re unfamiliar with Yes. >in domains where you have trouble judging the quality Sure, possibly. Kind of like how you think the news is accurate until you read a story that's in your field. But not necessarily. Might just be more "I don't know how do to <basic task> in <domain that I don't spend a lot of time in>", and LLMs are good at doing basic tasks.
moron4hire 1y ago
This feels like a parallel to the Gell-Mann amnesia effect. Recently, my company has been investigating AI tools for coding. I know this sounds very late to the game, but we're a DoD consultancy and one not traditional associated with software development. So, for most of the people in the company, they are very impressed with the AI's output. I, on the other hand, am a fairly recent addition to the company. I was specifically hired to be a "wildcard" in their usual operations. Which is too say, maybe 10 of us in a company of 3000 know what we're doing regarding software (but that's being generous because I don't really have visibility into half of the company). So, that means 99.7% of the company doesn't have the experience necessary to tell what good software development looks like. The stuff the people using the AI are putting out is... better than what the MilOps analysts pressed into writing Python-scripts-with-delusions-of-grandeur were doing before, but by no means what I'd call quality software. I have pretty deep experience in both back end and front end. It's a step above "code written by smart people completely inexperienced in writing software that has to be maintained over a lifetime", but many steps below, "software that can successfully be maintained over a lifetime".
- IX-103 1y ago
  Well, that's what you'd expect from an LLM. They're not designed to give you the best solution. They're designed to give you the most likely solution. Which means that the results would be expected to be average, as "above average" solutions are unlikely by definition. You can tweak the prompt a bit to skew the probability distribution with careful prompting (LLMs that are told to claim to be math PHDs are better at math problems, for instance), but in the end all of those weights in the model are spent to encode the most probable outputs. So, it will be interesting to see how this plays out. If the average person using AI is able to produce above average code, then we could end up in a virtuous cycle where AI continuously improves with human help. On the other hand, if this just allows more low quality code to be written then the opposite happens and AI becomes more and more useless.
  - leptons 1y ago
    I have no doubt which way it is going to go.
kannanvijayan 1y ago
I've been hacking on some somewhat systemsy rust code, and I've used LLMs from a while back (early co-pilot about a year ago) on a bunch of C++ systems code. In both of these cases, I found that just the smart auto-complete is a massive time-saver. In fact, it's more valuable to me than the interactive or agentic features. Here's a snippet of some code that's in one of my recent buffers: // The instruction should be skipped if all of its named // outputs have been coalesced away. if ! self.should_keep_instr(instr) { return; } // Non-dropped should have a choice. let instr_choice = choices.maybe_instr_choice(instr_ref) .expect("No choice for instruction"); self.pick_map.set_instr_choice( instr_ref, instr_choice.clone(), ); // Incref all named def inputs to the PIR choice. instr_choice.visit_input_defs(|input_def| { self.def_incref(input_def); }); // Decref all named def inputs to the SIR instr. instr.visit_inputs( |input_def| self.def_decref(input_def, sir_graph) ); The actual code _I_ wrote were the comments. The savings in not having to type out the syntax is pretty big. About 80% of the time in manual coding would have been that. Little typos, little adjustments to get the formatting right. The other nice benefit is that I don't have to trust the LLM. I can evaluate each snippet right there and typically the machine does a good job of picking out syntactic style and semantics from the rest of the codebase and file and applying it to the completion. The snippet, if it's not obvious, is from a bit of compiler backend code I'm working on. I would never have even _attempted_ to write a compiler backend in my spare time without this assistance. For experienced devs, autocomplete is good enough for massive efficiency gains in dev speed. I still haven't warmed to the agentic interfaces because I inherently don't trust the LLMs to produce correct code reliably, so I always end up reviewing it, and reviewing greenfield code is often more work than just writing it (esp now that autocomplete is so much more useful at making that writing faster).
- sgc 1y ago
  What exact tool are you using for your smart auto-complete?
  - kannanvijayan 1y ago
    Whatever copilot defaults to doing on vscode these days. I didn't configure it very much - just did the common path setup to get it working.
cguess 1y ago
This is exactly my experience as well. I've had agents write a bit of backend code, always small parts. I'm lucky enough to be experienced enough with code I didn't write to be able to quickly debug it when it fails (and it always fails from the first run). Like using AI to write a report, it's good for outlines, but the details are always seemingly random as far as quality. For frontend though? The stuff I really don't specialize in (despite some of my first html beginning on FrontPage 1997 back in 1997), it's a lifesaver. Just gotta be careful with prompts since so many front end frameworks are basically backend code at this point.
AstroBen 1y ago
This is good if front end is something you just need to get through. It's terrible if your work is moving to involve a lot of frontend - you'll never pick up the skills yourself
sysmax 1y ago
It works with low-level C/C++ just fine as long as you rigorously include all relevant definitions in the context window, provide non-obvious context (like the lifecycle of some various objects) and keep your prompts focused. Things like "apply this known algorithm to that project-specific data structure" work really well and save plenty of time. Things that require a gut feeling for how things are organized in memory don't work unless you are willing to babysit the model.
sensanaty 1y ago
As the fullstacker with a roughly 65/35 split BE/FE on the team who has to review this kinda stuff on the daily, there's nothing I dread more than a backender writing FE tickets and vice versa. Just last week I had to review some monstrosity of a FE ticket written by one of our backenders, with the comment of "it's 90% there, should be good to takeover". I had to throw out pretty much everything and rewrite it from scratch. My solution was like 150 lines modified, whereas the monstrous output of the AI was non-functional, ugly, a performance nightmare and around 800 lines, with extremely unhelpful and generic commit messages to the tune of "Made things great!!1!1!!". I can't even really blame them, the C-level craze and zeal for the AI shit is such that if you're not doing crap like this you get scrutinized and PIP'd. At least frontenders usually have some humility and will tell you they have no clue if it's a good solution or not, while BEnders are always for some reason extremely dismissive of FE work (as can be seen in this very thread). It's truly baffling to me

freetime2 1y ago

Here is the the methodology of the study: > To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. When AI is allowed, developers can use any tools they choose (primarily Cursor Pro with Claude 3.5/3.7 Sonnet—frontier models at the time of the study); when disallowed, they work without generative AI assistance. Developers complete these tasks (which average two hours each) while recording their screens, then self-report the total implementation time they needed. We pay developers $150/hr as compensation for their participation in the study. So it's a small sample size of 16 developers. And it sounds like different tasks were (randomly) assigned to the no-AI and with-AI groups - so the control group doesn't have the same tasks as the experimental group. I think this could lead to some pretty noisy data. Interestingly - small sample size isn't in the list of objections that the auther includes under "Addressing Every Objection You Thought Of, And Some You Didn’t". I do think it's an interesting study. But would want to see if the results could be reproduced before reading into it too much.

Tainnor 1y ago
The sample size isn't 16 developers, it's 246 issues.
- freetime2 1y ago
  So agree with that - but on the other hand surely the number of developers matters here? For example, if instead of 16 developers the study consisted of a single developer completing all 246 tasks with or without AI, and comparing the observed times to complete, I think most people would question the reproducibility and relevancy of the study?
  - Tainnor 1y ago
    It matters in the sense that it is unclear whether the findings generalise to other people. Which is a problem that a lot of studies, even with more participants, have because they may not have a diverse enough set of participants. But in terms of pure statistical validity, I don't think it matters.

budududuroiu 1y ago

I was surprised at how much better v0 was these days. I remember it yielding clunky UIs initially. I thought it was the model, but then I realised, v0 is carried by the shadcn UI library, not the intelligence of the model

JKCalhoun 1y ago

As others probably have experienced, I can only add that I am doing coding now I would have kicked down the road if I did not have LLM assistance. Example: using LeafletJS — not hard, but I didn't want to have to search all over to figure out how to use it. Example: other web page development requiring dropping image files, complicated scrolling, split-views, etc. In short, there are projects I have put off in the past but eagerly begin now that LLMs are there to guide me. It's difficult to compare times and productivity in cases like that.

georgemcbay 1y ago
This is pretty similar to my own experience using LLMs as a tool. When I'm working with platforms/languages/frameworks I am already deeply familiar with I don't think they save me much time at all. When I've tried to use them in this context they seem to save me a bunch of time in some situations, but also cost me a bunch of time in others resulting in basically a wash as far as time saved goes. And for me a wash isn't worth the long-term cost of losing touch with the code by not being the one to have crafted it. But when it comes to environments I'm not intimately familiar with they can provide a very easy on-ramp that is a much more pleasant experience than trying to figure things out through often iffy technical documentation or code samples.
timeon 1y ago
> search all over to figure out how to use it. Leaflet doc is single page document with examples you can copy-paste. There is page navogation at the top. Also ctrl/cmd+f and keyword seems quicker than writing the prompt.
- JKCalhoun 1y ago
  Nice. I'm afraid I simply assumed, like other "frameworks", it was going to entail wandering all over StackOverflow, etc. Still, when I simply told Claude that I wanted the pins to group together when zoomed out — it immediately knew I meant "clustering" and added the proper import to the top of the HTML file ... got it done.

tomcam 1y ago

What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.

journal 1y ago
what about the $ you make? isn't that an indicator? you've probably made more than me, so you are more successful while both of us might be doing the same thing.
- __MatrixMan__ 1y ago
  I don't think there's much of a correlation there.
  - __MatrixMan__ 1y ago
    A better measure would be how much work has been made unnecessary. The asymptote approached by software engineering is GDP=$0 because all problems are solved by maximally efficient automation. Never gonna happen, but progress on that path is a decent proxy for efficiency. Too often the job is about introducing problems that are good for some company's bottom line, but that's the opposite of efficiency.
- hammyhavoc 1y ago
  Productivity has zero to do with salary. Case in point: FOSS. Some of the most productive devs don't get paid by the big corps who make use of their open source projects, hence the constant urging of corps and people to sponsor projects they make money via.
- Ancalagon 1y ago
  In a vacuum I don’t believe pay alone is a very good indicator. What might be a better one is if someone has a history across their career of delivering working products to spec, doing this across companies and with increasing responsibility. This of course can only be determined after the fact.

hluska 1y ago

I’ve been around tech for a long time. At this point, I’ve lost count of how many hype cycles I’ve seen hit the “hold on, everything sucks” stage. Generative AI is seemingly at the hold on, everything sucks stage and it’s getting repetitive.

CaptainFever 1y ago
Trough of Disillusionment (followed by the Slope of Enlightenment and Plateau of Productivity): https://en.wikipedia.org/wiki/Gartner_hype_cycle https://en.wikipedia.org/wiki/Gartner_hype_cycle
- bluefirebrand 1y ago
  My bold prediction is that the Trough of Disillusionment for LLMs is going to be a very long stretch
deleted 1y ago
[deleted]

softwaredoug 1y ago

What if this is true? And then we as a developer community are focused on the wrong thing to increase productivity? Like what if by focusing on LLMs for productivity we just reinforce old-bad habits, and get into a local maxima... And even worse, what if being stuck with current so-so patterns, languages, etc means we don't innovate in language design, tooling, or other areas that might actually be productivity wins?

journal 1y ago
imagine having interstate highways built in one night you wake up and you have all these highways and roads and everyone is confused what they are and how to use them. using llm is the opposite of boiling frogs because you're not the leader writing, you're just suggesting... i just realized i might not know what im talking about.
__MatrixMan__ 1y ago
We were stuck near local maxima since before LLM's came on the scene. I figure the same concentration of innovators are gonna innovate, now LLM assisted, and the same concentration of best-practice folk are gonna best-practice--now LLM assisted. Local maxima might get sticker, but greener pastures will be found more quickly than ever. I expect it'll balance.
- softwaredoug 1y ago
  Honestly the biggest hindrance of developer productivity right now is probably perpetual, looming layoffs, not lacking AI, tools, programming languages, etc :)

raggi 1y ago

They averaged producing 47% more code on the AI tasks, but took only 20% more time. The report here biases over these considerations, but I’m left wondering: was the extra code superfluous or did this produce better structure / managed debt better? If that extra 47% of code translates to lower debt and more consistent throughput over the long term, I might take it, given how crushed projects get from debt. Anyway, it’s all hyperbole because there are massive statistical differences in the outcomes but no measures as to what they mean, but I’m sure they have meaning. That meaning matters a ton.

gpm 1y ago
Honestly my experience from using AI to code (primarily claude sonnet) is that that "extra 47%" is probably itself mostly tech debt. Places where the AI repeated itself instead of using a loop. Places where the AI wrote tests that don't actually test anything. Places where the AI failed to produce a simple abstraction and instead just kept doing the same thing by hand. Etc. AI isn't very good at being concise, in my experience. To the point of producing worse code. Which is a strange change from humans who might just have a habit of being too concise, but not by the same degree.
- raggi 1y ago
  Your response implies the ai produced code was landed without review. That’s a possible outcome but I would hope it’s unlikely to account for the whole group at this scale. We’re of course still lacking data.
  - trollbridge 1y ago
    I very much doubt that when individual programmers are producing significantly more code with the help of AI that somehow the review process simultaneously scales up to perform adequate review of all of that extra code. In my experience, review was inadequate back before we had AI spewing forth code of dubious quality. There's no reason to think it's any better now. An actually-useful AI would be one that would make reviews better, do them itself, or at least help me get through reviews faster.

kylecazar 1y ago

Now do a study that specifically gauges how useful an LLM (including smart tab completion) is for a frontend dev working in react/next/tailwind on everyday Jira tickets. These were maintainers of large open source projects. It's all relative. It's clearly providing massive gains for some and not as much for others. It should follow that it's benefit to you depends on who you are and what you are working on. It isn't black and white.

cheeze 1y ago
As a backend dev who owns a few internal crappy frontends, LLMs have been the best thing ever. Code quality isn't the top priority, I just need to plumb some data to an internal page at BigCorp.
- distalx 1y ago
  Could you share more about your process and how they specifically help you with your internal frontends? Any details would be great! Thanks!
franciscop 1y ago
It's a very well controlled study about... what the study claims to do. Yes, they didn't study a different thing, for _many_ reasons. Yes, we shouldn't haphazardly extrapolate to other parts of Engineering. But it looks like it's a good study nonetheless. There are some very good findings though, like how the devs thought they were sped up but they were actually slowed down.
- kylecazar 1y ago
  Agreed. My comment was anticipating the extrapolation part :)
timeon 1y ago
React and tailwind already made lot of tradeoffs to make it more ergonomic for developers. One would expect that LLMs could unlock lean and faster stack instead.

xarope 1y ago

I think this for me is the most worrying: "You can see that for AI Allowed tasks, developers spent less time researching and writing code". My analogy to this is seeing people spend time trying to figure out how to change colors, draw shapes in powerpoint, rather than focus on the content and presentation. So here, we have developers now focusing their efforts on correcting the AI output, rather than doing the research and improving their ability to deliver code in the future. Hmm...

hammyhavoc 1y ago
This has been my observation too. It's a tool for the lazy.
- yukai 1y ago
  laziness is a driving force of progress
  - anon15123 1y ago
    in what direction
    - baq 1y ago
      All of them.
      - badsectoracula 1y ago
        So the net result is we remain static? :-P
        baq 1y ago
        See also: WALL-E
- jack_pp 1y ago
  You can say the same about a printer. Or a kindle, oh you're too lazy to carry around 5 books with you?

dearilos 1y ago

I found that early and often code reviews can offset the reduction in productivity. A good code review process can fix this.

strangescript 1y ago

This entire concept hinges on AI not getting better. If you believe AI is going continue to get better at the current ~5-10% a month range, then hand waiving over developer productivity today is about the same thing as writing an article about the internet being a fad in 1999.

trashchomper 1y ago
On the flip side, why would I use AI today if it presents no immediate benefit. Why not wait 5 years and see if it becomes actually helpful.
- strangescript 1y ago
  better yet, wait 10, let me know how it goes
benrutter 1y ago
If they do improve at 5-10% a month then that'd definitely be true (tbh I'm not sure they are even improving at that rate now - 10% for a year would be 3x improvement with compounding). I guess the tricky bit is, nobody knows what the future looks like. "The internet is a fad" in 1999 hasn't aged well, but a lot of people touted 1960s AI, XML and 3d telivisions as things that'd be the tools in only a few years. We're all just guessing till then.

dismalaf 1y ago

I find LLMs are decent at regurgitating boilerplate. Basically the same kind of stuff you could google then copy-paste... AI chatbots, now that they have web access, are also good at going over documentation and save you a little time searching through the docs yourself. They're not great at business logic though, especially if you're doing anything remotely novel. Which is the difficult part of programming anyway. But yeah, to the average corporate programmer who needs to recreate the same internal business tool that every other company has anyway, it probably saves a lot of time.

trollbridge 1y ago
They're great at helping me figure out how to make something work with a poorly-documented, buggy framework, which is indeed a large fraction of my job, whether I like it or not.
handfuloflight 1y ago
This isn't true, and I know it by what I'm working on and sorry, I'm not at liberty to give more details. But I see how untrue this is, every working hour of every day.
- dismalaf 1y ago
  You say more details as if you gave any to begin with...
  - handfuloflight 1y ago
    Here's a hint: What I input for inference is not in the training data. But the model can generalize well enough to handle the task.
    - dismalaf 1y ago
      > What I input for inference is not in the training data ??? Like everyone else? This is why LLMs have large context windows. > But the model can generalize well enough to handle the task. Sounds like what you're doing is normal corporate business logic. If it's able to generalize it it's probably not particularly novel.

calrain 1y ago

I've been using Claude Code heavily for about 3 months now, and I'm pretty sure I'm between 10 and 20 times more productive while using it. How I measure performance is how many features I can implement in a given period of time. It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.

fuomag9 1y ago
Same, I’ve done stuff that should have taken me 2-3 weeks in days
- fourthark 1y ago
  I’ve done this without AI. The thing was not as hard as I thought it would be.
  - tracker1 1y ago
    Same, often a missing feature or library had become available since I last looked into a subject.
zsoltkacsandi 1y ago
I have exactly the same experience.
benreesman 1y ago
I find the swings to be wild, when you win with it, you win really big. But when you lose with it, it's a real bite out of your week too. And I think 10x to 20x has to be figurative right, you can do 20x by volume maybe, but to borrow an expression from Steve Ballmer, that's like measuring an airplane by kilograms. Someone already operating at the very limit of their abilities doing stuff that is for them high complexity, high cognitive load, detail intense, and tactically non-obvious? Even a machine that just handed you the perfect code can't 20x your real output, even if it gave you the source file at 20x your native sophistication you wouldn't be able to build and deploy it, let alone make changes to it. But even if it's the last 5-20% after you're already operating at your very limit and trying to hit your limit every single day is massive, it makes a bunch of stuff on the bubble go from "not realistic" to "we did that".

_jhk 1y ago

[dead]

camgunz 1y ago

I finally took the plunge and did a big chunk of work in Cursor. It was pretty ideal: greenfield but with a very relevant example to slightly modify (the example pulled events over HTTP as a server and I wanted it to pull events over Google pub/sub instead). Over IDK, 2-3 hours I got something that seemed on its face to work, but: - it didn't use the pub/sub API correctly - the 1 low-coverage test it generated didn't even compile (Go) - there were a bunch of small errors it got confused by--particularly around closures I got it to "90%" (again though it didn't at all work) with the first prompt, and then over something like a dozen more mostly got it to fix its own errors. But: - I didn't know the pub/sub API--I was relying on Cursor to do this correctly--and it totally submarined me - I had to do all the digging to get the test to compile - I had to go line by line and tell it to rewrite... almost everything I quit when I realized I was spending more time prompting it to fix things than it would take me to fully engage my brain and fix them myself. I also noticed that there was a strong pull to "just do one more prompt" rather than dig in and actually understand things. That's super problematic to me. Worse, this wasn't actually faster. How do I know that? The next day I did what I normally do: read docs and wrote it myself. I spent less time (I'm a fast typist and a Vim user) overall, and my code works. My experience matches pretty well w/ the results of TFA. --- Something I will say though is there is a lot of garbage stuff in tech. Like, I don't want to learn Terraform (again) just to figure out how to deploy things to production w/o paying a Heroku-like premium. Maybe I don't want to look up recursive CTEs again, or C function pointers, or spent 2 weeks researching a heisenbug I put into code for some silly reason AI would have caught immediately. I am _confident_ we can solve these things without boiling oceans to get AI to do it for us. But all this shit about how "I'm 20x more productive" is totally absurd. The only evidence we have of this is people just saying it. I don't think a 20x productivity increase is even imaginable. Overall productivity since 1950 is up 3.6x [0]. These people are asking us to believe they've achieved over 400 years of productivity gains in "3 months". Extraordinary claims require extraordinary evidence. My guess is either you were extremely unproductive before, or (like others are saying in the threads) in very small ways you're 20x more productive but most things are unaffected or even slower. [0]: https://fred.stlouisfed.org/series/OPHNFB https://fred.stlouisfed.org/series/OPHNFB

bonsai_bar 1y ago

Amaury-El 1y ago

The more I used it, the easier it became to skip over things I should have thought through myself. But looking back, the results weren’t always faster or better. Now I prefer to treat AI as a kind of challenger. It helps reveal the parts I haven't truly understood, rather than just speeding things up.

vouaobrasil 1y ago

AI could make me more productive, I know that for a fact. But, I don't want to be more productive because the tasks that could be automated with AI are those I find enjoyable. Not always in an intellectual sense, but in a meditative sense. And if I automated those away, I think I would become less human.

Fraterkes 1y ago

I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask. That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless. Conversely there's people who love ai and use it for everything, and since the majority of the stuff they ask about is fairly simple and well documented (eg "Write me some typescript"), they rarely have a negative experience.

aleph_minus_one 1y ago
I think there are also other aspects: - Some people simply ask a lot more questions than others (this ignores whether they like or dislike AI), i.e. some people simply prefer to find things out more by themselves, and thus also use other resources like Google or Stack Overflow as a last resort. So their questions to an AI will likely be more complicated, because they already found out the easy parts by themselves. - If I have to make the effort to explain to the AI in a sufficiently exhaustive way what I need (which I often have to do), I expect the answers of the AI to be really good. If it isn't, having explained my problem to the AI was simply a waste of time.
- deleted 1y ago
  [deleted]
- Tainnor 1y ago
  > I expect the answers of the AI to be really good. If it isn't, having explained my problem to the AI was simply a waste of time. I find the worst part to be when it doesn't correct flaws in my assumptions. For example, yesterday I asked it "what is the difference between these two Datadog queries"? And it replied something that was semi-correct, but it didn't discover the fundamental flaw - that the first one wasn't a valid query because of unbalanced parens. In fact, it turns out that the two strings (+ another one) would get concatenated and only then would it be a valid query. A simple "the first string is not a valid query because of a missing closing paren" would have saved a lot of time in trying to understand this, and I suspect that's what I would have received if I had prompted it with "what's the problem with this query" but LLMs are just too sycophantic to help with these things.

sheepfacts 1y ago

Perhaps is difficult to measure personal productivity in programming, but we can measure that we will run more slowly with 10 kg. in our backpack. I propose this procedure: The SWE selects 10 tasks and guesses some measure of their complexity (time to finish them) and then he randomly select 5 to be done with AI and the rest without. He performs them and finally calculates a deviation D. The deviation D = D_0 - D_1 where D_i = sum (real_time/guessed_time - 1), where D_0 is using AI and D_1 is without AI, the sign and magnitude of D measure respectively if the use of AI is beneficial or detrimental and the impact of using AI. Also, clipping individuals addends to be in the interval [-0.5,0.5] should avoid one bad guess dominating the estimation. Sorry if this is a trivial ideal but it is feasible and intuitively should provide useful information if the tasks are taken among the ones in which each initial guessing has small deviation. A filter should be applied to tasks in which scaffolding by AI surpass a certain relative threshold in case we are interested in generalizing our results to tasks in which scaffolding is not dominating time. It could happen that the impact of using AI depends of the task at hand, the capability of the SWE to pair programming with it, and of the LLM used, to such an extend that those factors were bigger that the average effect on a bag of tasks, in this case the large deviation from the mean makes any one parameter estimation void of useful information.

didibus 1y ago
That's pretty much what the study the article refers too did, and it found the use of AI was 19% slower.

titaniumrain 1y ago

history repeats itself - "horses are more efficient than cars" In addition, a study based on 16 devs is representative enough to draw this conclusion?

_heimdall 1y ago

I have never found a measure of programmer productivity that makes sense to me, but I can say that LLM coding tools are way more distracting to me than they are worth. They constantly guess at what I may type next, are often wrong, and pop in with suggestions breaking my mental flow and making me switch from the mindset of coding to the mindset of reviewing code.

Brendinooo 1y ago

The article brushed aside devs being terrible at estimates, but I dunno. I'm a frontend guy, been using Claude Code for a couple of weeks now. It's been able to speed up some boilerplate, it's sped up a lot of "naming is hard" conversations I like to have (but my coworkers probably don't, lol), it's enabled me to do a lot more stuff in my most recent project. But for a task or two I suspect that it has slowed me down. If I'm unable to articulate the problem well enough and the problem is hard enough you can go in circles for awhile. And I think the nature of "the right answer is just around the corner" makes it hard to timebox or find a specific point where you say "yup, time to ditch this and do it the old-fashioned way". There is a bit of a slot-machine effect here.

Jensson 1y ago
> But for a task or two I suspect that it has slowed me down Likely more, as it takes longer for you to activate your brain when your first thought is to ask an LLM rather than solve it yourself. Its like people reaching for a calculator to do 4+5, that doesn't make you faster or more accurate.
- Brendinooo 1y ago
  Depends on if the problem is 4+5 or something much more complex.

Fastjur 1y ago

> To compute the actual speedup – or, rather, slowdown! – provided by AI tools, the researchers compared the developers’ predictions of how long each task would take to the measured completion time. I'm sorry, but it feels to me like this research has only proven that developers tend to underestimate how long a task is supposed to take, with or without AI. In no way did they actually measure how much faster a specific task was when performed with and without AI?

didibus 1y ago
What I understand they did is. You have two tasks: - Task 1 - 3h - Task 2 - 1h You ask the dev to estimate both. Then you randomly tell the dev, ok do Task 1 without AI, and Task 2 with AI. Then you measure the actual time it took. Their estimate for AI task missed the mark by 19%, but those without AI were done 20% faster then estimated. At the time of estimating they didn't know if the task would need to be done with AI or not.
- Fastjur 1y ago
  Right, that makes more sense indeed.

anupj 1y ago

[flagged]

b0a04gl 1y ago

[dead]

greatgib 1y ago

Really a great piece of work. At the opposite of the usual studies posted here. With the headline we can easily guess that the study should be flawed, with the sample not representative, or developers not expert enough with the AI or so, and then, they give a very well done list of all valable objections with arguments about why they don't think that contradict the study. That replied to all the questions I could have had in the end.

SkyPuncher 1y ago

One thing I find frustrating with these conversations is the _strict_ focus on single-task productivity. Arguably, on a single coding task, I don't really move that much faster. However, I have much, much more brain capacity left both while coding and when I'm done coding. This has two knock on effects: 1. Most simply, I'm productive for longer. Since LLMs are doing a lot of the heavy lifting, my brain doesn't have to constantly think. This is especially important in time periods where I'd previously have too little mental energy to think deeply about code. 2. I can do other things while coding. Well, right now, Cursor is cycling on a simple task while I type this. Most days, though, I'm responding to customers, working on documentation/planning, or doing some other non-coding task that's critical to my workflows. This is actually where I find my biggest productivity gains. Instead of coding THEN X, I can now do coding WITH X.

bluefirebrand 1y ago
> I can do other things while coding Context shifting while trying to code seems like a bad idea to me Maybe you're some incredible multi -tasking genius able to change tasks rapidly without losing any of the details, but I suspect if most people tried this workflow they would produce worse code and also whatever their other output is would be low quality too
- SkyPuncher 1y ago
  For me, the most time consuming parts of coding don’t require deep meaningful thought. It’s often things like stepping through code to figure out where something breaks (answer is often obvious once you find it) or iterating through broken specs or simply playing with a library/interface enough to figure out how it actually works (vs how’s it’s documented) This type of stuff is great to have an LLM tackle while I work on something else.

bhaktatejas922 1y ago

I have never felt so disconnected from the findings of a study

bhaktatejas922 1y ago

A lot of this stems from friction in brittle process around applying edits or latency from full file rewrites. The bar to beat is time it takes for human intelligence + Cursor Tab to complete a task, but in practice the bar becomes "feeling" like its faster