6 ms·

> You get the LLM to draft some code for you that’s 80% complete/correct. > You tweak the last 20% by hand. > How much of a productivity increase is that? Wel

• __loam 2y ago
  It's fascinating watching people in ostensibly leadership positions in our industry continue to fundamentally misunderstand how productivity works in knowledge work.
  • ezekg 2y ago
    They never understood. AI is the new Mythical Man-Month.
    • saghm 2y ago
      The example I used to hear to demonstrate this type of logical fallacy is that even though one woman can make a baby in 9 months, having eight other women help doesn't make the baby available after a single month. I guess this might need to be updated to "one woman and eight LLMs can't make a baby in one month".
  • seabombs 2y ago
    Totally. It used to be that LoC was widely accepted as a a terrible metric to measure dev productivity by, but when talking about AI code-gen tools it's like it's all that matters.
• remram 2y ago
  The problem is that now, when your colleagues/contributors push out shitty code that you have to fix... you don't know if they even tried or just put in bad code from an LLM.
  • deleted 2y ago
    [deleted]
  • mintplant 2y ago
    Yep, this is a hell I've come to know over the past year. You get people trying to build things they really have no business building, not without at least doing their own research and learning first, but the LLM in their editor lets them spit out something that looks convincing enough to get shipped, unless someone with domain experience looks at it more closely and realizes it's all wrong. It's not the same as bodged-together code from Stack Overflow because the LLM code is better at getting past the sniff test, and magical thinking around generative AI leads people to put undue trust in it. I don't think we've collectively appreciated the negative impact this is going to have on software quality going forward.
  • rad_gruchalski 2y ago
    Yeah, but you can teach your colleagues to get better. An LLM will give you the same crap every time.
    • remram 2y ago
      You missed my point, there is no use teaching those who are cheating. And there is no use learning if you don't get called out.
      • rad_gruchalski 2y ago
        Thanks tor pointing that out. I missed that, indeed. That’s a stellar point.
      • BeetleB 2y ago
        > You missed my point, there is no use teaching those who are cheating. It's a very invalid point. Perhaps in your life you've never encountered people who cheat and later change their ways, but it's fairly common for them to change their ways.
        • 2y ago
• bongodongobob 2y ago
  Except you saved yourself all that mental effort. If it takes the same amount of time and I don't need to put 110% focus in, it's a win.
  • fiddlerwoaroof 2y ago
    Verifying code is much more mental effort than writing it because you have to reconstruct the data structure and the logic of the code in order to show that the code you’re reading is correct.
  • yoyohello13 2y ago
    You also absorbed no knowledge of the problem domain. I'm pretty positive on LLMs but with the caveat that the programmer still should be an active participant in the development of the solution.
    • bongodongobob 2y ago
      I gave it the knowledge of the domain. I never said the programmer shouldn't be an active participant either. There's no way around that.
  • phyzome 2y ago
    No, now I get to be a code reviewer instead of a writer, and that's a lot more mental energy for me.
• ezekg 2y ago
  This is pretty much it. I don't trust code produced by AI, tbh, because it almost always has logical errors, or it ignored part of the prompt so it's subtly wrong. I really like AI for prototyping and sketching, but then I usually take that prototype, maybe write some solid tests, and then write the real implementation.
  • energy123 2y ago
    If the code these LLMs produce has logical errors and bugs, I have formed the opinion that this is mostly user error. LLMs produce great code if they're used properly. By that I mean: strictly control the maximum level of complexity that you're asking the LLMs to solve. GPT-4o can only handle rudimentary coding tasks. So I only ask it rudimentary questions. And it's rarely wrong or bugged. o1-2024-12-17 is much better, it can handle "mid" complexity tasks, it's usually never wrong on those, but a step above and it will output buggy code. This is subjective and not perfect, but when you use the models more you can get a sense for where it starts falling apart and stay below that threshold. This means chunking your problem into multiple prompts, where each chunk is nearing the maximum level of complexity that the model can handle.
    • onemoresoop 2y ago
      The way many LLms are coded is to not ask questions where clarification is needed but just eagerly produce something and fill in the gaps.
• dudeinjapan 2y ago
  [dead]
• saghm 2y ago
  Yeah, this is almost literally the 80-20 rule[1]; often getting the first 80% of something done takes 20% of the work, and getting the last 20% takes 80% of the work. Debugging the last 20% is already the hard part for most things, and as you mention, it's going to be a lot harder debugging LLM-generated code. [1]: https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle
  • gregw2 2y ago
    re: debugging I dont disagree but here's another spin on what you observed. Kernighan's law says you have to be twice as intelligent/skilled when debugging the code as you were when writing it. A LLM centric corollary would be -don't have the LLM write code at the boundary of its capabilities in your domain. It's a little hard these days to know where that line lies with the rate of improvements but perhaps is an intuition people are still new at with LLMs.
• deleted 2y ago
  [deleted]
• golergka 2y ago
  I've built a pet project with LLMs recently. It's rough around the edges, with bugs, incomplete functionality and docs, but I've built it around 4 times as fast as I would have done it myself. https://github.com/golergka/hn-blog-recommender https://github.com/golergka/hn-blog-recommender Basically 95% of the code is written by Aider. I still had to do QA, make high-level decisions, review code it wrote, assign tasks and tell it when to refactor stuff. I would never use this workflow to write a program where absolute correctness is top priority, like a radiation therapy controller. But from this point on, I'll never waste time building pet projects, small web app and utilities myself.
  • __loam 2y ago
    And you'll be a worse programmer for it because you won't have the reps.
    • nradov 2y ago
      Programming is a means to an end for most of us. If you want to become an expert programmer for the sake of the craft then go for it, but most would benefit more from learning more about the business side. If you're building the wrong functionality then it doesn't matter how good your code is.
      • __loam 2y ago
        It's fascinating seeing this take on Hackernews of all places. I'm probably one of the most mercenary programmers you'll meet but this take is cynical even to me. If you have a skill that you use professionally and get paid hundreds of thousands of dollars to do, maybe you should cultivate and maintain it, idk. You'll probably get paid more and work less if you know what you're doing as opposed to being yet another disposable I'll handle the business side guy.
    • golergka 2y ago
      I've been programming since 95. I have crafted beautiful handwritten codebases with much more complicated requirements. But at this point, I see code quality not as an end goal, but one of the trade offs.
• onethumb 2y ago
  I've deployed code with Claude that was >100X faster, all-in (verification, testing, etc) than doing it myself. (I've been writing code professionally and successfully for >30 years). >100X isn't typical, I'd guess typical in my case is >2X, but it's happened at large scale and saved me months of work in this single instance. (Years of work overall)
  • machinekob 2y ago
    If you wirte 100x faster code you could probably automate almost all of it away as it seems to be super trivial and already resolved problems. I also use Claude for my coding a lot, but i’m not sure about real gains if it even give me noticeable speed improvement as i’m in a loop of “waiting” and fixing bugs that LLM made still it is super useful for writing big doc strings and smaller tests maybe if i focus on some basic tasks like classical backend or frontend it’ll be more useful
• Diggsey 2y ago
  Using AI is like trying to build a product with only extremely junior devs. You can get something working fairly quickly but sooner rather than later it's going to collapse in on itself and there won't be anyone around capable of fixing it. Moving past that requires someone with more experience to figure out what they did and eventually fix it, which ends up being far more expensive than correcting things along the way. The error rate needs to come down significantly from where it's currently at, and we'd also need AIs that cover each others weaknesses - at the moment you can't just run another AI to fix the 20% of errors the first one generated because they tend to have overlapping blind spots. Having said all that, I think AI is hugely useful as a programming tool. Here are the places I've got most value: - Better IDE auto-complete. - Repetetive re-factors that are beyond what's possible to do with a regex. For example, replaing one function with another whose usage is subtly different. LLMs can crack out those changes across a codebase and it's easy to review. - Summarizing and extracting intent from dense technical content. I'm often implementing some part of a web standard, and being able to quickly ask an AI questions, and have it give references back to specific parts of the standard is super useful. - Bootstrapping tests. The hardest part of writing tests in my experience is writing the first one - there's always a bit of boilerplate that's annoying to write. The AI is pretty good at generating a smoke test and then you can use that a basis to do proper test coverage. I'm sure there are more. The point is that it's currently incredibly valuable as a tool, but it's not yet capable of being an agent.
  • onemoresoop 2y ago
    It seems like nobody cares about the code quality, it just has to work until the conpany is acquired
• yapyap 2y ago
  why dont they get another LLM to do the last 20%, are they stupid?
• johnfn 2y ago
  The thing that everyone seems to miss here is it's not surprising which code falls into the 80% or 20%. I can guess if an AI will write something bug-free with almost 100% accuracy. For the remaining 20%, whatever, it gives me a fine enough starting point.
• deleted 2y ago
  [deleted]
• vunderba 2y ago
  Agreed. To add to this: The article is using a rather poor comparison because it makes the assumption that all sections of a program take an equal amount of time to write. An LLM might be able to hammer out the unit tests, boilerplate react logic, and the express backend in 5 minutes, but if it collapses on the "20%" that you need actual assistance with, then the productivity boost is significantly less than suggested.
• taneq 2y ago
  To debug code that doesn't work, you generally have to fully understand the goal and the code. To write code that does work, you generally have to fully understand the goal and what code to use. So unless you just flat-out don't know what code to use (ie. libraries, architecture, approach etc.) it's going to be no faster and probably less satisfying to fix LLM code as it would be to just do it yourself. Where LLMs do shine in coding (in my limited experience) though is discovery. If you need to write something in a mostly unfamiliar language and ecosystem they can be great for getting off the ground. I'm still not convinced that the mental gear shifting between 'student asking questions' and 'researcher/analyst/coder mode' is worth the slightly more targeted responses vs. just looking up tutorials, but it's more useful scenario.
• edanm 2y ago
  > Except, it’s probably four times as hard to verify the 80% right code as it is to just write the code yourself in the first place. Why would that be true in general? You think it just so happens that a new tool gives an X speedup in ability but costs an exactly equivalent X slowdown in other places?
  • fiddlerwoaroof 2y ago
    No, the claim is just that it’s harder to read and verify existing code than to write new code. Depending on the level of certainty you want, verifying code is much more expensive than writing code, which is why there are so many bugs and why PR review is, imo, practically worthless. Switching the job of programming from designing and writing code to reading and verifying code isn’t a good trade-off in terms of making programming easier. And my first speei node with AI tools has done nothing to change my mind on this point.