7 ms·

> I could have written it but it would have taken me about 3x longer when all is said and done. Really does not sound like that from your description. It sound

> I could have written it but it would have taken me about 3x longer when all is said and done. Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself. Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not?

giantg2 1y ago
"Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself." And if this is true, you will have to coach AI each time whereas a person should advance over time.
- rolisz 1y ago
  And then they skip to another job for more money, and you start again with a new hire.
  - Avicebron 1y ago
    Thankfully after many generations of human interactions and complex analysis of group dynamics, we've found a solution. It's called 'don't be an asshole' and 'pay people competitively'. edit: because people are stupid, 'competitively' in this sense isn't some theoretical number pulled from an average, it's 'does this person feel better off financially working with you than others around them who don't work with you, and is is this person meeting their own personal financial goals through working with you'?
    - wiseowise 1y ago
      They do have a point. Why waste a time on person who will always need more money over time, rather than invest in AI? Not only you don’t need to please every hire, your seniors will be more thankful too, because they will get linearly faster with time.
      - smsm42 1y ago
        Outside of working for Antropic etc., there's no way you can make an LLM better at anything. You can train a junior though.
        victorbjorklund 1y ago
        You can def provide better context etc.
        wiseowise 1y ago
        You absolutely can. It's a skill like operating IDE, CLI, etc. Junior is a person, not your personal assistant like LLM.
    - faangguyindia 1y ago
      The person paying and the one responsible for coaching others usually aren't same
    - binary132 1y ago
      The elephant in this particular room is that there are a tiny handful of employers that have so much money that they can and do just pay whatever amount is more than any of their competitors can possibly afford.
      - giantg2 1y ago
        That shouldn't be a big deal since they're a finite portion of the market. You should have a robust enough model to handle people leaving, including unavoidable scenarios like retirement and death.
    - thfuran 1y ago
      The common corporate policy of making it harder to give raises than to increase starting salaries for new hires is insane.
      - SOLAR_FIELDS 1y ago
        Is it insane? Makes perfect sense. Employee has way less leverage at raise time. It’s all about leverage. It sucks, but that is the reality
    - ponector 1y ago
      Mentor usually has no power over the compensation for the mentee. Also it is never a policy to pay competitively for the existing employees, only for the new hires.
  - giantg2 1y ago
    That's not a bad thing. It means you've added one more senior to the societal pool. A lot of the talent problems today are due to companies not wanting to train and focusing on cheap shortcut options like outsourcing or H1B
- raincole 1y ago
  At least you can ask AI to summarize a AGENT.md or something and it will read it diligently next time. As for humans, they might not have the motivation technical writing skill to document what they learnt. And even if they did, the next person might not have the patience to actually read it.
  - giantg2 1y ago
    I find the summaries to be helpful. However, I find some of the detailed points to lack a deep understanding of technical points and their importance.
  - ay 1y ago
    "Read diligently" - that’s a very optimistic statement. I can not count how many times Claude (LLM I am most familiar with, I had it write probably about 100KLOC in the past few months) explicitly disobeyed what was written in the instructions. Also, a good few times, if it were a human doing the task, I would have said they both failed to follow the instructions and lied about it and attempted to pretend they didn’t. Luckily their lying abilities today are primitive, so it’s easy to catch.
    - derefr 1y ago
      > Also, a good few times, if it were a human doing the task, I would have said they both failed to follow the instructions and lied about it and attempted to pretend they didn’t. It's funny. Just yesterday I had the experience of attending a concert under the strong — yet entirely mistaken — belief that I had already been to a previous performance of the same musician. It was only on the way back from the show, talking with my partner who attended with me (and who had seen this musician live before), trying to figure out what time exactly "we" had last seen them, with me exhaustively listing out recollections that turned out to be other (confusingly similar) musicians we had seen live together... that I finally realized I had never actually been to one of this particular musician's concerts before. I think this is precisely the "experience" of being one of these LLMs. Except that, where I had a phantom "interpolated" memory of seeing a musician I had never actually seen, these LLMs have phantom actually-interpolated memories of performing skills they have never actually themselves performed. Coding LLMs are trained to replicate pair-programming-esque conversations between people who actually do have these skills, and are performing them... but where those conversations don't lay out the thinking involved in all the many implicit (thinking, probing, checking, recalling) micro-skills involved in actually performing those skills. Instead, all you get in such a conversation thread is the conclusion each person reaches after applying those micro-skills. And this leads to the LLM thinking it "has" a given skill... even though it doesn't actually know anything about "how" to execute that skill, in terms of the micro-skills that are used "off-screen" to come up with the final response given in the conversation. Instead, it just comes up with a prediction for "what someone using the skill" looks like... and thinks that that means it has used the skill. Even after a hole is poked in its use of the skill, and it realizes it made a mistake, that doesn't dissuade it from the belief that it has the given skill. Just like, even after I asked my partner about the show I recall us attending, and she told me that that was a show for a different (but similar) musician, I still thought I had gone to the show. It took me exhausting all possibilities for times I could have seen this musician before, to get me to even hypothesize that maybe I hadn't. And it would likely take similarly exhaustive disproof (over hundreds of exchanges) to get an LLM to truly "internalize" that it doesn't actually have a skill it believed itself to have, and so stop trying to use it. (If that meta-skill is even a thing that LLMs have ever learned from their training data — which I doubt. And even if they did, you'd be wasting 90% of a Transformer's context window on this. Maybe something that's worth keeping in mind if we ever switch back to basing our LLMs on RNNs with true runtime weight updates, though!)
- mensetmanusman 1y ago
  The AI in this example is 1/100 the cost.
  - cratermoon 1y ago
    For now, not including externalities.
  - gnerd00 1y ago
    that is absolutely false - the capital and resources used to create these things are societal scale. An individual consumer is not paying that cost at this time.
    - mensetmanusman 1y ago
      That only proves the point. If something increases the value of someone’s time by 5% and 500,000,000 people are affected by it, the cost will collapse. These models are only going to get better and cheaper per watt.
      - devmor 1y ago
        > These models are only going to get better and cheaper per watt. What do you base this claim on? They have only gotten exponentially more expensive for decreasing gain so far - quite the opposite of what you say.
    - victorbjorklund 1y ago
      You can make the same argument about humans. The employeer doesnt pay the full cost and time to create the worker from an embryo to a senior dev.
      - devmor 1y ago
        Unless you are advocating for executing developers when they are no longer capable of working, that’s a bit of a non sequitur. Humans aren’t tools.
nicce 1y ago
> Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself. Even if you do it by yourself, you need to do the same thinking and iterative process by yourself. You just get the code almost instantly and mostly correctly, if you are good at defining the initial specification.
- fsloth 1y ago
  This. You _have_ to write the spec. The result is that instead of spending X units of time on spec and THEN y units of time on coding, you get the whole thing in x units of time AND you have a spec. The trick is knowning where the particular LLM sucks. I expect in a short amount of time there is no productivity gain but when you start to understand the limitations and strengths - holey moley.
  - skydhash 1y ago
    > The result is that instead of spending X units of time on spec and THEN y units of time on coding, you get the whole thing in x units of time AND you have a spec. It's more like x units of time thinking and y units of times coding, whereas I see people spend x/2 thinking, x typing the specs, y correcting the specs, and y giving up and correcting the code.
    - fsloth 1y ago
      Sure! That's inefficient. I know just how I work and I've been writing the type of programs I do for quite many years. And I know what would take me normally a week takes me few days at best.
    - nicce 1y ago
      But if you can’t define the specs, you can’t write the code without LLM either?
  - smsm42 1y ago
    Unless you realize no LLM is good at what you need and you just wasted weeks of time walking in circles.
aljimbra 1y ago
My buggy executive function frequently gets in the way of putting code to screen. You know how hacker news has that lil timeout setting to pseudo force you to disengage from it? AI made it so I don't need anything like that. It is digital Adderall.
veidr 1y ago
It is in many ways much like coaching a n00b, but a n00b that can do 10 hours of n00b work in 10 minutes (or, 2 minutes). That's a significant difference. There are a lot of tasks that can be done by a n00b with some advice, especially when you can say "copy the pattern when I did this same basic thing here and here". And there are a lot of things a n00b, or an LLM, can't do. The study you reference was real, and I am not surprised — because accurately gauging the productivity win, or loss, obtained by using LLMs in real production coding workflows is also not junior stuff.
athrowaway3z 1y ago
> Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not? On a tangent; that study is brought up a lot. There are some issues with it, but I agree with the main takeaway to be weary of the feeling of productivity vs actual productivity. But most of the time its brought up by AI skeptics, that conveniently gloss over the fact it's about averages. Which, while organizationally interesting, is far less interesting than to discover what is and isn't currently possible at the tail end by the most skillful users.
- kaydub 1y ago
  Not to mention the study doesn't really show a lack of productivity and they include some key caveats in it outlining how they think productivity increases using LLMs
- oceanplexian 1y ago
  Engineers have always been terrible at measuring productivity. Building a new internal tool or writing a bunch of code is not necessarily productive. Productivity is something that creates business value. In that sense an engineer who writes 10 lines of code but that code solves a $10M business problem or allows the company to sign 100 new customers may be the most productive engineer in your organization.
- sarchertech 1y ago
  The key insight from the study is that even the users that did see an increase in productivity overestimated that increase. Taken along with the dozens of other studies that show that humans are terrible at estimating how long it will take them to complete task, you should be very skeptical when someone says an LLM makes them x% more productive. There’s no reason to think that the most skillful LLM users are not overestimating productivity benefits as well.
tarsinge 1y ago
Sure it’s a lot of work, but the noob in question has all the internet knowledge and can write multiples times faster than a human for a fraction of the costs. This is not about an individual being more productive, this is about business costs. Long term we should still hire and train juniors obviously, but short term there is lot of pressure to not do it as it makes no sense financially. Study or not the reality is there is not much difference in productivity between a senior with a cursor license and a senior and a junior that needs heavy guidance.
- skydhash 1y ago
  Code is a liability. You always want less of it. Typing faster does not particularly help. Unless the tool is verbose, then you fix the tool.
ants_everywhere 1y ago
There was one study that said that in a specific setting and was amplified heavily on forums by anti-AI people. There have been many more studies showing productivity gains across a variety of tasks that preceded that one. That study wasn't necessarily wrong about the specific methodology they had for onboarding people to use AI. But if I remember correctly it was funded by an organization that was slightly skeptical of AI.
- JohnMakin 1y ago
  which studies show this?
  - simonw 1y ago
    Here are some from the last few months: AI coding assistant trial: UK public sector findings report: https://www.gov.uk/government/publications/ai-coding-assistant-trial/ai-coding-assistant-trial-uk-public-sector-findings-report https://www.gov.uk/government/publications/ai-coding-assista... - UK government. "GDS ran a trial of AI coding assistants (AICAs) across government from November 2024 to February 2025. [...] Trial participants saved an average of 56 minutes a working day when using AICAs" Human + AI in Accounting: Early Evidence from the Field: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240924 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240924 - "We document significant productivity gains among AI adopters, including a 55% increase in weekly client support and a reallocation of approximately 8.5% of accountant time from routine data entry toward high-value tasks such as business communication and quality assurance." OECD: The effects of generative AI on productivity, innovation and entrepreneurship: https://www.oecd.org/en/publications/the-effects-of-generative-ai-on-productivity-innovation-and-entrepreneurship_b21df222-en.html https://www.oecd.org/en/publications/the-effects-of-generati... - "Generative AI has proven particularly effective in automating tasks that are well-defined and have clear objectives, notably including some writing and coding tasks. It can also play a critical role for skill development and business model transformation, where it can serve as a catalyst for personalised learning and organisational efficiency gains, respectively [...] However, these potential gains are not without challenges. Trust in AI-generated outputs and a deep understanding of its limitations are crucial to leverage the potential of the technology. The reviewed experiments highlight the ongoing need for human expertise and oversight to ensure that generative AI remains a valuable tool in creative, operational and technical processes rather than a substitute for authentic human creativity and knowledge, especially in the longer term.".
micromacrofoot 1y ago
It's this, but 1000 times faster — that's the difference. It's sending a noob away to follow your exact instructions and getting results back in 10 seconds instead of 10 hours. I don't have to worry about managing the noob's emotions or their availability, I can tell the LLM to try 3 different approaches and it only takes a few minutes... I can get mad at it and say "fuck it I'll do this part myself", the LLM doesn't have to be reminded of our workflow or formatting (I just tell the LLM once) I can tell it that I see a code smell and it will usually have an idea of what I'm talking about and attempt to correct, little explanation needed The LLM can also: do tons of research in a short amount of time, traverse the codebase and answer questions for me, etc it's a noob savant It's no replacement for a competent person, but it's a very useful assistant
ludicrousdispla 1y ago
LLMs make two people more productive, the person that uses the LLM, and then the person that cleans up the mess.
pkilgore 1y ago
You aren't wrong in the coaching but, but feedback loops are orders of magnitude faster. It takes an LLM 2-20 minutes to give me the next stage of output not 1-2 days (week?). As a result, I have higher context the entire time so my side of the iteration is maybe 10x faster too.
fluidcruft 1y ago
True but the n00b is very fast. A lot of coaching is waiting for the n00b to perform tasks and meta things about motivation. These LLM are extremely fast and eager to work. I don't need a study to tell me that five projects that have been stuck in slow plodding along waiting for me to ever have time or resources for nearly ten years. But these are now nearing completion after only two months of picking up Claude Code. And with high-quality implementations that were feverdreams. My background is academic science not professional programming though and the output quality and speed of Claude Code is vastly better than what grad students generate. But you don't trust grad student code either. The major difference here is that suggestions for improvement loop in minutes rather than weeks or months. Claude will get the science wrong, but so do grad students. (But sure technically they are not finished yet ... but yeah)
- dingnuts 1y ago
  LLMs might make you feel faster (which helps with motivation!) and help with some of the very easy stuff but the critical part of your anecdote is that you haven't actually completed the extra work. The projects are only "NEARING" completion. I think that's very telling.
  - victorbjorklund 1y ago
    If the easy things are done faster you xan spend more time on the hard stuff. No need to spend 2 hours on making the UI for the MVP when an AI can make a decent UI in 2 min. Means you have 2 hours more to spend on the hard stuff.
    - SpicyLemonZest 1y ago
      Unless, as is often the case in my experience, the hard stuff consists largely of fixing bugs and edge cases in your implementation of the easy stuff. I've seen multiple people already end up forced back to the drawing board because their "easy stuff" AI implementation had critical flaws they only realized after they thought they were done. It's hard to prove counterfactuals, but I'm pretty confident they would have gotten it right the first time if they hadn't used AI, they're not bad engineers.
peteforde 1y ago
I am so tired of this style of "don't believe your lying eyes" conjecture. I'm a career coder and I used LLMs primarily to rapidly produce code for domains that I don't have deep experience in. Instead of spending days or weeks getting up to speed on an SDK I might need once, I have a pair programmer that doesn't check their phone or need to pick up their kids at 4:30pm. If you don't want to use LLMs, nobody is forcing you. Burning energy trying to convince people to whom the benefits of LLMs are self-evident many times over that they are imagining things is insulting the intelligence of everyone in the conversation.
- vlovich123 1y ago
  Correct. In areas you yourself are a junior engineer, you’ll be more effective with an LLM at tackling that area maybe. It’s also surprisingly effective at executing refactors.
  - peteforde 1y ago
    I'm not sure which one of us is ultimately more hung up on titles in this context, but I would push back and say that when someone with 30+ years experience tackling software problems delegates navigating the details of an API to an LLM, that is roughly the most "senior developer" moment of the day. Conflating experience and instinct with knowing everything isn't just false equivalency, it's backwards.
    - vlovich123 1y ago
      I really don’t know what I said that was such an emotional trigger for you. All I said is that it’s an accelerant for you when you leave your domain. Like for example I’m a systems engineer. I hate coding UIs but with the LLM I can pump out a UI quickly and this was true both for web code and a GUI I built with dioxus. The UI code was also cleaner because I had some sense of how it should be structured and asked the AI to cleanup that structure. But ultimately it did most of the work in response to high level prompts and I picked and chose what to review line by line vs vibe coding. That’s what I mean - by myself it would have taken me easily 10x longer if not worse because UI coding for me is a slog + there’s nuances about reactive coding + getting started is also a hurdle. The output of the code was still high quality because I knew when the LLM wasn’t making the choices I wanted it to make.
lumost 1y ago
Anecdotally, on green field projects where you are exploring a new domain - it’s an insanely productive experience. On mundane day to day tasks it probably takes more time, but feels like less mental bandwidth. Coding at full throttle is a very intensive task that requires deep focus. There are many days that I simply don’t have that in me.
kaydub 1y ago
Everyone using that study to prove LLMs are bad hasn't actually read the study.
- Hendrikto 1y ago
  I don’t think anyone said anything about “good VS bad”, just about actual VS reported productivity impacts.
dpflan 1y ago
The study you are alluding to is this one by METR (Model Evaluation & Threat Research): Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - https://arxiv.org/abs/2507.09089 https://arxiv.org/abs/2507.09089 """ Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). """
TeMPOraL 1y ago
> Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not? Curious about specifics of this study. Because in general, how one feels is critical to productivity. It's hard to become more productive when the work is less and less rewarding. The infamous "zone" / "flow state" involves, by its very definition, feeling of increasing productivity being continuously reinforced on a minutes-by-minutes level. Etc.
- lkey 1y ago
  perceived result: 20% faster actual result: 20% slower link: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
- bluefirebrand 1y ago
  Personally this might explain why I feel so demoralized by LLMs I hate the experience of trying to write code with them. I like to just type my thoughts into the files. I hate trying to involve the LLM, even as a search. I want my search to feel like looking up references not having a conversation with a robot Overall, for me, the whole experience of trying to code with LLMs is both frustrating and unrewarding And it definitely doesn't seem more efficient or faster either
moomoo11 1y ago
I built a new service recently. It has about a dozen or so endpoints, facilitating real time messaging. It took me about 4 hours to build it out, fully tested with documentation and examples and readme. About two hours were spent setting up the architecture and tests. About 45 min to an hour setting up a few of the endpoints. The rest were generated by CC. FWIW it is using layers and SRP to the max. Everything is simple and isolated, easy to test. I think if I had a contractor or employee do this they would have coasted for a week and still fucked it up. Adding ridiculous complexity or just fucking up. The nice thing about AI tools is you need less people. Most people are awful at their jobs, anyone can survive a few years and call themselves senior. Most teams are only successful because of the 1 or 2 guys who pull 150% while the others are barely doing 80%.

giantg2 1y ago

"Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself." And if this is true, you will have to coach AI each time whereas a person should advance over time.

rolisz 1y ago
And then they skip to another job for more money, and you start again with a new hire.
- Avicebron 1y ago
  Thankfully after many generations of human interactions and complex analysis of group dynamics, we've found a solution. It's called 'don't be an asshole' and 'pay people competitively'. edit: because people are stupid, 'competitively' in this sense isn't some theoretical number pulled from an average, it's 'does this person feel better off financially working with you than others around them who don't work with you, and is is this person meeting their own personal financial goals through working with you'?
  - wiseowise 1y ago
    They do have a point. Why waste a time on person who will always need more money over time, rather than invest in AI? Not only you don’t need to please every hire, your seniors will be more thankful too, because they will get linearly faster with time.
    - smsm42 1y ago
      Outside of working for Antropic etc., there's no way you can make an LLM better at anything. You can train a junior though.
      - victorbjorklund 1y ago
        You can def provide better context etc.
      - wiseowise 1y ago
        You absolutely can. It's a skill like operating IDE, CLI, etc. Junior is a person, not your personal assistant like LLM.
  - faangguyindia 1y ago
    The person paying and the one responsible for coaching others usually aren't same
  - binary132 1y ago
    The elephant in this particular room is that there are a tiny handful of employers that have so much money that they can and do just pay whatever amount is more than any of their competitors can possibly afford.
    - giantg2 1y ago
      That shouldn't be a big deal since they're a finite portion of the market. You should have a robust enough model to handle people leaving, including unavoidable scenarios like retirement and death.
  - thfuran 1y ago
    The common corporate policy of making it harder to give raises than to increase starting salaries for new hires is insane.
    - SOLAR_FIELDS 1y ago
      Is it insane? Makes perfect sense. Employee has way less leverage at raise time. It’s all about leverage. It sucks, but that is the reality
  - ponector 1y ago
    Mentor usually has no power over the compensation for the mentee. Also it is never a policy to pay competitively for the existing employees, only for the new hires.
- giantg2 1y ago
  That's not a bad thing. It means you've added one more senior to the societal pool. A lot of the talent problems today are due to companies not wanting to train and focusing on cheap shortcut options like outsourcing or H1B
raincole 1y ago
At least you can ask AI to summarize a AGENT.md or something and it will read it diligently next time. As for humans, they might not have the motivation technical writing skill to document what they learnt. And even if they did, the next person might not have the patience to actually read it.
- giantg2 1y ago
  I find the summaries to be helpful. However, I find some of the detailed points to lack a deep understanding of technical points and their importance.
- ay 1y ago
  "Read diligently" - that’s a very optimistic statement. I can not count how many times Claude (LLM I am most familiar with, I had it write probably about 100KLOC in the past few months) explicitly disobeyed what was written in the instructions. Also, a good few times, if it were a human doing the task, I would have said they both failed to follow the instructions and lied about it and attempted to pretend they didn’t. Luckily their lying abilities today are primitive, so it’s easy to catch.
  - derefr 1y ago
    > Also, a good few times, if it were a human doing the task, I would have said they both failed to follow the instructions and lied about it and attempted to pretend they didn’t. It's funny. Just yesterday I had the experience of attending a concert under the strong — yet entirely mistaken — belief that I had already been to a previous performance of the same musician. It was only on the way back from the show, talking with my partner who attended with me (and who had seen this musician live before), trying to figure out what time exactly "we" had last seen them, with me exhaustively listing out recollections that turned out to be other (confusingly similar) musicians we had seen live together... that I finally realized I had never actually been to one of this particular musician's concerts before. I think this is precisely the "experience" of being one of these LLMs. Except that, where I had a phantom "interpolated" memory of seeing a musician I had never actually seen, these LLMs have phantom actually-interpolated memories of performing skills they have never actually themselves performed. Coding LLMs are trained to replicate pair-programming-esque conversations between people who actually do have these skills, and are performing them... but where those conversations don't lay out the thinking involved in all the many implicit (thinking, probing, checking, recalling) micro-skills involved in actually performing those skills. Instead, all you get in such a conversation thread is the conclusion each person reaches after applying those micro-skills. And this leads to the LLM thinking it "has" a given skill... even though it doesn't actually know anything about "how" to execute that skill, in terms of the micro-skills that are used "off-screen" to come up with the final response given in the conversation. Instead, it just comes up with a prediction for "what someone using the skill" looks like... and thinks that that means it has used the skill. Even after a hole is poked in its use of the skill, and it realizes it made a mistake, that doesn't dissuade it from the belief that it has the given skill. Just like, even after I asked my partner about the show I recall us attending, and she told me that that was a show for a different (but similar) musician, I still thought I had gone to the show. It took me exhausting all possibilities for times I could have seen this musician before, to get me to even hypothesize that maybe I hadn't. And it would likely take similarly exhaustive disproof (over hundreds of exchanges) to get an LLM to truly "internalize" that it doesn't actually have a skill it believed itself to have, and so stop trying to use it. (If that meta-skill is even a thing that LLMs have ever learned from their training data — which I doubt. And even if they did, you'd be wasting 90% of a Transformer's context window on this. Maybe something that's worth keeping in mind if we ever switch back to basing our LLMs on RNNs with true runtime weight updates, though!)
mensetmanusman 1y ago
The AI in this example is 1/100 the cost.
- cratermoon 1y ago
  For now, not including externalities.
- gnerd00 1y ago
  that is absolutely false - the capital and resources used to create these things are societal scale. An individual consumer is not paying that cost at this time.
  - mensetmanusman 1y ago
    That only proves the point. If something increases the value of someone’s time by 5% and 500,000,000 people are affected by it, the cost will collapse. These models are only going to get better and cheaper per watt.
    - devmor 1y ago
      > These models are only going to get better and cheaper per watt. What do you base this claim on? They have only gotten exponentially more expensive for decreasing gain so far - quite the opposite of what you say.
  - victorbjorklund 1y ago
    You can make the same argument about humans. The employeer doesnt pay the full cost and time to create the worker from an embryo to a senior dev.
    - devmor 1y ago
      Unless you are advocating for executing developers when they are no longer capable of working, that’s a bit of a non sequitur. Humans aren’t tools.

nicce 1y ago

> Really does not sound like that from your description. It sounds like coaching a noob, which is a lot of work in itself. Even if you do it by yourself, you need to do the same thinking and iterative process by yourself. You just get the code almost instantly and mostly correctly, if you are good at defining the initial specification.

fsloth 1y ago
This. You _have_ to write the spec. The result is that instead of spending X units of time on spec and THEN y units of time on coding, you get the whole thing in x units of time AND you have a spec. The trick is knowning where the particular LLM sucks. I expect in a short amount of time there is no productivity gain but when you start to understand the limitations and strengths - holey moley.
- skydhash 1y ago
  > The result is that instead of spending X units of time on spec and THEN y units of time on coding, you get the whole thing in x units of time AND you have a spec. It's more like x units of time thinking and y units of times coding, whereas I see people spend x/2 thinking, x typing the specs, y correcting the specs, and y giving up and correcting the code.
  - fsloth 1y ago
    Sure! That's inefficient. I know just how I work and I've been writing the type of programs I do for quite many years. And I know what would take me normally a week takes me few days at best.
  - nicce 1y ago
    But if you can’t define the specs, you can’t write the code without LLM either?
- smsm42 1y ago
  Unless you realize no LLM is good at what you need and you just wasted weeks of time walking in circles.

aljimbra 1y ago

My buggy executive function frequently gets in the way of putting code to screen. You know how hacker news has that lil timeout setting to pseudo force you to disengage from it? AI made it so I don't need anything like that. It is digital Adderall.

veidr 1y ago

It is in many ways much like coaching a n00b, but a n00b that can do 10 hours of n00b work in 10 minutes (or, 2 minutes). That's a significant difference. There are a lot of tasks that can be done by a n00b with some advice, especially when you can say "copy the pattern when I did this same basic thing here and here". And there are a lot of things a n00b, or an LLM, can't do. The study you reference was real, and I am not surprised — because accurately gauging the productivity win, or loss, obtained by using LLMs in real production coding workflows is also not junior stuff.

athrowaway3z 1y ago

> Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not? On a tangent; that study is brought up a lot. There are some issues with it, but I agree with the main takeaway to be weary of the feeling of productivity vs actual productivity. But most of the time its brought up by AI skeptics, that conveniently gloss over the fact it's about averages. Which, while organizationally interesting, is far less interesting than to discover what is and isn't currently possible at the tail end by the most skillful users.

kaydub 1y ago
Not to mention the study doesn't really show a lack of productivity and they include some key caveats in it outlining how they think productivity increases using LLMs
oceanplexian 1y ago
Engineers have always been terrible at measuring productivity. Building a new internal tool or writing a bunch of code is not necessarily productive. Productivity is something that creates business value. In that sense an engineer who writes 10 lines of code but that code solves a $10M business problem or allows the company to sign 100 new customers may be the most productive engineer in your organization.
sarchertech 1y ago
The key insight from the study is that even the users that did see an increase in productivity overestimated that increase. Taken along with the dozens of other studies that show that humans are terrible at estimating how long it will take them to complete task, you should be very skeptical when someone says an LLM makes them x% more productive. There’s no reason to think that the most skillful LLM users are not overestimating productivity benefits as well.

tarsinge 1y ago

Sure it’s a lot of work, but the noob in question has all the internet knowledge and can write multiples times faster than a human for a fraction of the costs. This is not about an individual being more productive, this is about business costs. Long term we should still hire and train juniors obviously, but short term there is lot of pressure to not do it as it makes no sense financially. Study or not the reality is there is not much difference in productivity between a senior with a cursor license and a senior and a junior that needs heavy guidance.

skydhash 1y ago
Code is a liability. You always want less of it. Typing faster does not particularly help. Unless the tool is verbose, then you fix the tool.

ants_everywhere 1y ago

There was one study that said that in a specific setting and was amplified heavily on forums by anti-AI people. There have been many more studies showing productivity gains across a variety of tasks that preceded that one. That study wasn't necessarily wrong about the specific methodology they had for onboarding people to use AI. But if I remember correctly it was funded by an organization that was slightly skeptical of AI.

JohnMakin 1y ago
which studies show this?
- simonw 1y ago
  Here are some from the last few months: AI coding assistant trial: UK public sector findings report: https://www.gov.uk/government/publications/ai-coding-assistant-trial/ai-coding-assistant-trial-uk-public-sector-findings-report https://www.gov.uk/government/publications/ai-coding-assista... - UK government. "GDS ran a trial of AI coding assistants (AICAs) across government from November 2024 to February 2025. [...] Trial participants saved an average of 56 minutes a working day when using AICAs" Human + AI in Accounting: Early Evidence from the Field: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240924 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240924 - "We document significant productivity gains among AI adopters, including a 55% increase in weekly client support and a reallocation of approximately 8.5% of accountant time from routine data entry toward high-value tasks such as business communication and quality assurance." OECD: The effects of generative AI on productivity, innovation and entrepreneurship: https://www.oecd.org/en/publications/the-effects-of-generative-ai-on-productivity-innovation-and-entrepreneurship_b21df222-en.html https://www.oecd.org/en/publications/the-effects-of-generati... - "Generative AI has proven particularly effective in automating tasks that are well-defined and have clear objectives, notably including some writing and coding tasks. It can also play a critical role for skill development and business model transformation, where it can serve as a catalyst for personalised learning and organisational efficiency gains, respectively [...] However, these potential gains are not without challenges. Trust in AI-generated outputs and a deep understanding of its limitations are crucial to leverage the potential of the technology. The reviewed experiments highlight the ongoing need for human expertise and oversight to ensure that generative AI remains a valuable tool in creative, operational and technical processes rather than a substitute for authentic human creativity and knowledge, especially in the longer term.".

micromacrofoot 1y ago

It's this, but 1000 times faster — that's the difference. It's sending a noob away to follow your exact instructions and getting results back in 10 seconds instead of 10 hours. I don't have to worry about managing the noob's emotions or their availability, I can tell the LLM to try 3 different approaches and it only takes a few minutes... I can get mad at it and say "fuck it I'll do this part myself", the LLM doesn't have to be reminded of our workflow or formatting (I just tell the LLM once) I can tell it that I see a code smell and it will usually have an idea of what I'm talking about and attempt to correct, little explanation needed The LLM can also: do tons of research in a short amount of time, traverse the codebase and answer questions for me, etc it's a noob savant It's no replacement for a competent person, but it's a very useful assistant

ludicrousdispla 1y ago

LLMs make two people more productive, the person that uses the LLM, and then the person that cleans up the mess.

pkilgore 1y ago

You aren't wrong in the coaching but, but feedback loops are orders of magnitude faster. It takes an LLM 2-20 minutes to give me the next stage of output not 1-2 days (week?). As a result, I have higher context the entire time so my side of the iteration is maybe 10x faster too.

fluidcruft 1y ago

True but the n00b is very fast. A lot of coaching is waiting for the n00b to perform tasks and meta things about motivation. These LLM are extremely fast and eager to work. I don't need a study to tell me that five projects that have been stuck in slow plodding along waiting for me to ever have time or resources for nearly ten years. But these are now nearing completion after only two months of picking up Claude Code. And with high-quality implementations that were feverdreams. My background is academic science not professional programming though and the output quality and speed of Claude Code is vastly better than what grad students generate. But you don't trust grad student code either. The major difference here is that suggestions for improvement loop in minutes rather than weeks or months. Claude will get the science wrong, but so do grad students. (But sure technically they are not finished yet ... but yeah)

dingnuts 1y ago
LLMs might make you feel faster (which helps with motivation!) and help with some of the very easy stuff but the critical part of your anecdote is that you haven't actually completed the extra work. The projects are only "NEARING" completion. I think that's very telling.
- victorbjorklund 1y ago
  If the easy things are done faster you xan spend more time on the hard stuff. No need to spend 2 hours on making the UI for the MVP when an AI can make a decent UI in 2 min. Means you have 2 hours more to spend on the hard stuff.
  - SpicyLemonZest 1y ago
    Unless, as is often the case in my experience, the hard stuff consists largely of fixing bugs and edge cases in your implementation of the easy stuff. I've seen multiple people already end up forced back to the drawing board because their "easy stuff" AI implementation had critical flaws they only realized after they thought they were done. It's hard to prove counterfactuals, but I'm pretty confident they would have gotten it right the first time if they hadn't used AI, they're not bad engineers.

peteforde 1y ago

I am so tired of this style of "don't believe your lying eyes" conjecture. I'm a career coder and I used LLMs primarily to rapidly produce code for domains that I don't have deep experience in. Instead of spending days or weeks getting up to speed on an SDK I might need once, I have a pair programmer that doesn't check their phone or need to pick up their kids at 4:30pm. If you don't want to use LLMs, nobody is forcing you. Burning energy trying to convince people to whom the benefits of LLMs are self-evident many times over that they are imagining things is insulting the intelligence of everyone in the conversation.

vlovich123 1y ago
Correct. In areas you yourself are a junior engineer, you’ll be more effective with an LLM at tackling that area maybe. It’s also surprisingly effective at executing refactors.
- peteforde 1y ago
  I'm not sure which one of us is ultimately more hung up on titles in this context, but I would push back and say that when someone with 30+ years experience tackling software problems delegates navigating the details of an API to an LLM, that is roughly the most "senior developer" moment of the day. Conflating experience and instinct with knowing everything isn't just false equivalency, it's backwards.
  - vlovich123 1y ago
    I really don’t know what I said that was such an emotional trigger for you. All I said is that it’s an accelerant for you when you leave your domain. Like for example I’m a systems engineer. I hate coding UIs but with the LLM I can pump out a UI quickly and this was true both for web code and a GUI I built with dioxus. The UI code was also cleaner because I had some sense of how it should be structured and asked the AI to cleanup that structure. But ultimately it did most of the work in response to high level prompts and I picked and chose what to review line by line vs vibe coding. That’s what I mean - by myself it would have taken me easily 10x longer if not worse because UI coding for me is a slog + there’s nuances about reactive coding + getting started is also a hurdle. The output of the code was still high quality because I knew when the LLM wasn’t making the choices I wanted it to make.

lumost 1y ago

Anecdotally, on green field projects where you are exploring a new domain - it’s an insanely productive experience. On mundane day to day tasks it probably takes more time, but feels like less mental bandwidth. Coding at full throttle is a very intensive task that requires deep focus. There are many days that I simply don’t have that in me.

kaydub 1y ago

Everyone using that study to prove LLMs are bad hasn't actually read the study.

Hendrikto 1y ago
I don’t think anyone said anything about “good VS bad”, just about actual VS reported productivity impacts.

dpflan 1y ago

The study you are alluding to is this one by METR (Model Evaluation & Threat Research): Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - https://arxiv.org/abs/2507.09089 https://arxiv.org/abs/2507.09089 """ Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). """

TeMPOraL 1y ago

> Wasn’t there a study that said that using LLMs makes people feel more productive while they actually are not? Curious about specifics of this study. Because in general, how one feels is critical to productivity. It's hard to become more productive when the work is less and less rewarding. The infamous "zone" / "flow state" involves, by its very definition, feeling of increasing productivity being continuously reinforced on a minutes-by-minutes level. Etc.

lkey 1y ago
perceived result: 20% faster actual result: 20% slower link: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
bluefirebrand 1y ago
Personally this might explain why I feel so demoralized by LLMs I hate the experience of trying to write code with them. I like to just type my thoughts into the files. I hate trying to involve the LLM, even as a search. I want my search to feel like looking up references not having a conversation with a robot Overall, for me, the whole experience of trying to code with LLMs is both frustrating and unrewarding And it definitely doesn't seem more efficient or faster either

moomoo11 1y ago

I built a new service recently. It has about a dozen or so endpoints, facilitating real time messaging. It took me about 4 hours to build it out, fully tested with documentation and examples and readme. About two hours were spent setting up the architecture and tests. About 45 min to an hour setting up a few of the endpoints. The rest were generated by CC. FWIW it is using layers and SRP to the max. Everything is simple and isolated, easy to test. I think if I had a contractor or employee do this they would have coasted for a week and still fucked it up. Adding ridiculous complexity or just fucking up. The nice thing about AI tools is you need less people. Most people are awful at their jobs, anyone can survive a few years and call themselves senior. Most teams are only successful because of the 1 or 2 guys who pull 150% while the others are barely doing 80%.