5 ms·

is it really more efficient to have an LLM generate code, then review that code, fix errors and spend some time to fully understand it? I wish there were tangib

is it really more efficient to have an LLM generate code, then review that code, fix errors and spend some time to fully understand it? I wish there were tangible stats and metrics around this. Is it really more efficient than just writing the code yourself, but using LLMs to look up things or demo solutions?

tptacek 1y ago
Yes, but the bar for skepticism is higher than that, because LLMs also compile code and catch errors, and generate and run tests; compile errors and assertion failures are just more prompts to an LLM agent.
- what 1y ago
  Citation needed. Here is copilot “fixing” a failing test: https://github.com/dotnet/runtime/commit/fe173fc8f44dbd0a9ed909383f0dc17bc9a36d35 https://github.com/dotnet/runtime/commit/fe173fc8f44dbd0a9ed... It rewrote some comments, changed the test name and added extra assertions to the test. Baby sitting something like that seems like an absolute waste of time.
  - BeetleB 1y ago
    You want a citation for things so many people are doing daily with LLMs? Just because they can't fix most failures doesn't mean they can't fix many.
    - what 1y ago
      It’s a failure that it created and when told to fix it did this. It’s beyond bad.
      - BeetleB 1y ago
        > It's a failure that it created and when told to fix it did this. It’s beyond bad. No one's disputing this was bad. People are merely claiming it can also be good. I've dealt with plenty of humans this bad - it's not an argument that humans can't program.
        deleted 1y ago
        [deleted]
        callc 1y ago
        It seems like the underlying issue is trust.competent programmer - even a junior - and trust them to finish the task correctly. It might take multiple tries, and they may ask for clarification, but since they’re human, we trust they are intelligent. There are some people who fall into the bucket that we can’t trust them to finish the task correctly, or within a time frame or level of effort on our part to make the task offloading exercise have a positive benefit. If we view LLMs in the same light, IMO currently they fall into “not trust” category to really give they a task and trust them to finish it correctly, with us being confident we don’t really need to understand their implementation. If one day LLMs or some other solution reaches that point, then it definitely won’t look like a bubble, but a real revolution.
        BeetleB 1y ago
        Very well put. The trick is to do either of the following: 1. Find simpler tasks for which the trust in LLMs is high. 2. Give tasks to the LLMs that have a very low cost to verify (even when the task is not simple) - particularly one off scripts. I once had a colleague who was in the "not trust" bucket for the work we were doing. So we found something he was good at that was a pain for me to do, and re-assigned him to do those things and take that burden off of us. In the last few months I've had the LLM solve (simple) problems via code that had been in my head for years. At any point I could have done them, but they were a chore. If the LLM failed for one of these tasks - it's not a big deal - not much time was lost. But they tend to succeed fairly often, because they are simple tasks. I almost never let the LLM write production code, because of the extra burden that you and others allude to. But I do let it write code I rely on in my personal life, because frankly I tend to write pretty poor code for my personal use - I can't justify the time it would take to write things well - life is too busy. I welcome the code quality I get from Sonnet or Gemini 2.5 Pro. That's my point in this thread. Writing code is a pretty diverse discipline, and many are dismissing it simply because it doesn't do one particular use case (high quality production code) well. I didn't take LLM coding seriously until I found well respected, well known SW engineers speak positively about them. Then I tried it and ... oh wow. People dismissing them is dismissing not only a lot of average developers' reality, but also a lot of experts' daily reality. Just look at the other submission: https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/ https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-... He used an LLM to find a security vulnerability in the kernel. To quote him: > Before I get into the technical details, the main takeaway from this post is this: with o3 LLMs have made a leap forward in their ability to reason about code, and if you work in vulnerability research you should start paying close attention. If you’re an expert-level vulnerability researcher or exploit developer the machines aren’t about to replace you. In fact, it is quite the opposite: they are now at a stage where they can make you significantly more efficient and effective. If you have a problem that can be represented in fewer than 10k lines of code there is a reasonable chance o3 can either solve it, or help you solve it.
  - motorest 1y ago
    > It rewrote some comments, changed the test name and added extra assertions to the test. Baby sitting something like that seems like an absolute waste of time. I've worked with real flesh and blood developers who did the exactly same thing. At least with LLMs we don't have to jump into a 1h long call to discuss the changes.
- lolinder 1y ago
  When used that way they also regularly get into loops where they lose track of what they were supposed to do. The last time I set Cursor on something without watching it very very closely it spun for a while fixing tests and when it finally stopped and I looked what it had done it had coded special cases in to pass the specific failing tests in a way that didn't generalize at all to the actual problem. Another recent time I had to pull the plug on it installing a bunch of brand new dependencies that it decided would somehow fix the failing tests. It had some kind of complete rewrite planned. Claude Code is even worse when it gets into this mode because it'll do something totally absurd like that and then at the end you have to `git reset` and you're also on the hook for the $5 of tokens that it managed to spend in 5 minutes. I still find them useful, but it takes a lot of practice to figure out when they'll be useful and when they'll be a total waste of time.
  - tptacek 1y ago
    It happens to me every once in awhile, but I'm not sure why I would care. I usually set it off on some question and go tab away to something else while it flails. When I come back, I have a better-than-average shot at a workable solution, which is a science fiction result. When I first began programming as a teenager, one of the mental hurdles I had to get over was asking the computer to "too much"; like, I would feel bad writing a nested loop --- that can't possibly be the right answer! What a chore for the computer! It didn't take me too long to figure out that was the whole point of computers. To me, it's the same thing with LLMs spinning on something. Who gives a shit? It's not me wasting that time.
    - kasey_junk 1y ago
      I’m now at the point where I tell it to do something _and then drive to Wisconsin_ giving feedback at the Marz cheese castle and continuing my trip.
      - tptacek 1y ago
        I feel like you might know where I'm coming from being confused at people's reaction to this stuff. This is science fiction. I think it's just not sinking in with people. If I could bet on this, I would bet everything I could on "skill with LLMs" being the high order bit of being an effective software developer 5 years from now.
viraptor 1y ago
Depends on the code, but often yes. The less you care about that specific result, the more efficient it is. One-off tools under 2k lines where you can easily verify the result? Why would I not generate that and save time for more interesting stuff?
- layer8 1y ago
  One-off tools seem to be more ops then dev, to me.
  - jiggawatts 1y ago
    I’ve used Gemini Pro 2.5 to generate code syntax rewrite tools from me. I asked it to use a compiler SDK that I know from experience is fiddly and frustrating to use. It gave me a working tool in about 10 minutes, while I was actively listening to a meeting. Not all scripts are “ops”!
  - viraptor 1y ago
    I haven't heard the "scripting is not programming" or similar take since newsgroups. It's really time to let it die.
    - layer8 1y ago
      That’s not what I wrote. It’s just that in my dev work I only rarely have the need for one-off tools.
      - sagarm 1y ago
        Do you not debug, optimize, analyze ...? Copilot is especially valuable for throwaway (or nearly throwaway) log parsing/analysis, for example.
        layer8 1y ago
        I do, but I have all the software I need for that. I may use LLMs in that context, but for the actual analysis, not for generating one-off tools.
Aurornis 1y ago
Lately I look for opportunities to have the LLM do some easy small work while I go work on some harder task in parallel. I've also tried asking the LLM to come up with a proposed solution while I work on my own implementation at the same time. LLMs can also be much faster if a task requires some repetitive work. When I recognize a task like that, I try coding the first version and then ask the LLM to follow my pattern for the other areas where I need to repeat the work.
- zackify 1y ago
  Agree with each of these points so much! That’s why I really like copilot agent and codex right now. Even more parallel stuff and from my phone when I’m just thinking of ideas.
- mwcampbell 1y ago
  How did we end up here, just accepting that some coding tasks require repetitive work, and turning to a probabilistic text synthesizer that requires massive training data to automate that? We're having this discussion on a site whose founder famously wrote, over 20 years ago, that succinctness is power, and even wrote this site in a programming language that he designed to take that principle as far as he could. Now, two decades later, why have we so completely retreated from that dream? I bear some responsibility for this, since I was one of the people who basically said, in the 2010s, that we should just give up and use popular languages like JavaScript because they're popular. I regret that now.
  - Aeolun 1y ago
    Some things are better with more code. Sets of data in particular (strongly typed). Sometimes you need to modify those sets of data and it doesn’t require enough work to write a whole script for, but you still don’t really want to spend the time manually modifying everything. LLM’s are really nice in those instances.
    - klabb3 1y ago
      Plain data in all glory but I’d rather (deterministically) generate and modify it than having big blobs checked in. If I have too much copy pasted data I often forget to modify until the test case runs and I realize it. This creates a false sense of confidence since there could be tests that pass and nobody ever checks they’re wrong. Essentially, minimize the number of human steps to verify it looks right which typically means optimizing for the least amount of code. Flashback to when I committed a suite of tests in Python that were indented one tab too much, resulting in them not running at all. This passed code review (on a FAANG company) and was discovered months later from an unrelated bug. The point is even unit tests have a very human element to them.
simonw 1y ago
Depends how good you are at reading and reviewing code. If you're already good at that LLMs are a huge productivity boost.
motorest 1y ago
> is it really more efficient to have an LLM generate code, then review that code, fix errors and spend some time to fully understand it? The answer is always "it depends". There are some drudge work tasks that are brilliantly done by LLMs, such as generating unit tests or documentation. Often the first attempt is not great, but iterating over them is so fast that you can regenerate everything from scratch a dozen times before you spend as much time as you would do if you wrote them yourself. It also depends on what scope you're working on. Small iterations have better results than grand redesigns. Context is also critical. if your codebase is neatly organized with squeaky clean code then LLMs generate better recommendations. If your codebase is a mess with inconsistent styles in spaghetti code then your prompts tend to generate more of the same.
max_on_hn 1y ago
(Disclaimer: I built and sell a product around that workflow) It often is, if you pick the right tasks (and more tasks fall into that bucket every few weeks). You can get a simple but fully-working app out of a single prompt, though quality varies widely unless you’re very specific. Once you have a codebase, agent output quality comes down to architecture and tests. If you have a scalable architecture with well-separated concerns, a solid integration test harness with examples, and good documentation (features, stack, procedures, design constraints), then getting the exact change you want is a matter of how well you can articulate what you want. One more asterisk, the development environment has to support the agent: like a human, agents work well with compiler feedback, and better with testing tools and documentation/internet access (yes my agents have these). I use CheepCode to work on itself, but I am still building up the test library and preview environments to de-risk merging non-trivial PRs that I haven’t pulled down and run locally. I also use it to work on other apps that I'm building, and since those are far more self-contained / easier to test, I get much better results there. If you want to put less effort into describing what you want, have a chat with an AI to generate tickets. Then paste those tickets into Linear and let CheepCode agents rip through them. I’ve got tooling in the works that will make that much easier, but I can only be in so many places at once as a bootstrapped founder :-)
teemur 1y ago
If you can identify blocks of code you need to write that are easy to define reasonably well, easy to review/verify that it is written correctly but still burdensome to actually write, LLMs are your new best friend. I don't know about how other people think/write, but I seem to have a lot of that kind of stuff on my table. The difficult part to outsource to LLMs is how to connect these easy blocks, but luckily thats the part I find fun in coding, not so much writing the boring simple stuff.
jsosshbfn 1y ago
LLMs tend to type a little faster than humans... so for straightforward code, yes?
mattlondon 1y ago
I frequently use it to assemble the boiler plate tedious stuff. E.g. add a new test file and wire up all the build rules etc, then implement tests x y z. Or the other day I had to create an input form in an Angular app and I just asked for the 7 or 8 fields I wanted and it did it and wired it all up for me (both template and Typescript). It can do those sorts of things in a few seconds and saves me maybe only 15-20 minutes but the mental relief of me not having to do it is great, even if the time saving is relatively low. The other thing is sometimes just writing out method signatures with all the right types and conversions/casts etc between types/interfaces/classes etc when I can't be bothered to do all the look ups myself ("Create a method here called foo that accepts a Bar instance and converts it to Baz. Return type should be Quux - add a conversion from Baz to a new Quux instance before the final return - use the builder pattern and map the constant magic-strings in blah.ts to appropriate enum values in Quux." Etc etc and then I write the logic in the middle) Again not a huge time saving but it mentally lightens the load and keeps you concentrating on the problem rather than the minutia
IshKebab 1y ago
In many cases yes. In many cases no. Overall, it can save you a lot of time even accounting for time wasted when you give it more than it can handle and you give up and do it all yourself.

tptacek 1y ago

Yes, but the bar for skepticism is higher than that, because LLMs also compile code and catch errors, and generate and run tests; compile errors and assertion failures are just more prompts to an LLM agent.

what 1y ago
Citation needed. Here is copilot “fixing” a failing test: https://github.com/dotnet/runtime/commit/fe173fc8f44dbd0a9ed909383f0dc17bc9a36d35 https://github.com/dotnet/runtime/commit/fe173fc8f44dbd0a9ed... It rewrote some comments, changed the test name and added extra assertions to the test. Baby sitting something like that seems like an absolute waste of time.
- BeetleB 1y ago
  You want a citation for things so many people are doing daily with LLMs? Just because they can't fix most failures doesn't mean they can't fix many.
  - what 1y ago
    It’s a failure that it created and when told to fix it did this. It’s beyond bad.
    - BeetleB 1y ago
      > It's a failure that it created and when told to fix it did this. It’s beyond bad. No one's disputing this was bad. People are merely claiming it can also be good. I've dealt with plenty of humans this bad - it's not an argument that humans can't program.
      - deleted 1y ago
        [deleted]
      - callc 1y ago
        It seems like the underlying issue is trust.competent programmer - even a junior - and trust them to finish the task correctly. It might take multiple tries, and they may ask for clarification, but since they’re human, we trust they are intelligent. There are some people who fall into the bucket that we can’t trust them to finish the task correctly, or within a time frame or level of effort on our part to make the task offloading exercise have a positive benefit. If we view LLMs in the same light, IMO currently they fall into “not trust” category to really give they a task and trust them to finish it correctly, with us being confident we don’t really need to understand their implementation. If one day LLMs or some other solution reaches that point, then it definitely won’t look like a bubble, but a real revolution.
        BeetleB 1y ago
        Very well put. The trick is to do either of the following: 1. Find simpler tasks for which the trust in LLMs is high. 2. Give tasks to the LLMs that have a very low cost to verify (even when the task is not simple) - particularly one off scripts. I once had a colleague who was in the "not trust" bucket for the work we were doing. So we found something he was good at that was a pain for me to do, and re-assigned him to do those things and take that burden off of us. In the last few months I've had the LLM solve (simple) problems via code that had been in my head for years. At any point I could have done them, but they were a chore. If the LLM failed for one of these tasks - it's not a big deal - not much time was lost. But they tend to succeed fairly often, because they are simple tasks. I almost never let the LLM write production code, because of the extra burden that you and others allude to. But I do let it write code I rely on in my personal life, because frankly I tend to write pretty poor code for my personal use - I can't justify the time it would take to write things well - life is too busy. I welcome the code quality I get from Sonnet or Gemini 2.5 Pro. That's my point in this thread. Writing code is a pretty diverse discipline, and many are dismissing it simply because it doesn't do one particular use case (high quality production code) well. I didn't take LLM coding seriously until I found well respected, well known SW engineers speak positively about them. Then I tried it and ... oh wow. People dismissing them is dismissing not only a lot of average developers' reality, but also a lot of experts' daily reality. Just look at the other submission: https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/ https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-... He used an LLM to find a security vulnerability in the kernel. To quote him: > Before I get into the technical details, the main takeaway from this post is this: with o3 LLMs have made a leap forward in their ability to reason about code, and if you work in vulnerability research you should start paying close attention. If you’re an expert-level vulnerability researcher or exploit developer the machines aren’t about to replace you. In fact, it is quite the opposite: they are now at a stage where they can make you significantly more efficient and effective. If you have a problem that can be represented in fewer than 10k lines of code there is a reasonable chance o3 can either solve it, or help you solve it.
- motorest 1y ago
  > It rewrote some comments, changed the test name and added extra assertions to the test. Baby sitting something like that seems like an absolute waste of time. I've worked with real flesh and blood developers who did the exactly same thing. At least with LLMs we don't have to jump into a 1h long call to discuss the changes.
lolinder 1y ago
When used that way they also regularly get into loops where they lose track of what they were supposed to do. The last time I set Cursor on something without watching it very very closely it spun for a while fixing tests and when it finally stopped and I looked what it had done it had coded special cases in to pass the specific failing tests in a way that didn't generalize at all to the actual problem. Another recent time I had to pull the plug on it installing a bunch of brand new dependencies that it decided would somehow fix the failing tests. It had some kind of complete rewrite planned. Claude Code is even worse when it gets into this mode because it'll do something totally absurd like that and then at the end you have to `git reset` and you're also on the hook for the $5 of tokens that it managed to spend in 5 minutes. I still find them useful, but it takes a lot of practice to figure out when they'll be useful and when they'll be a total waste of time.
- tptacek 1y ago
  It happens to me every once in awhile, but I'm not sure why I would care. I usually set it off on some question and go tab away to something else while it flails. When I come back, I have a better-than-average shot at a workable solution, which is a science fiction result. When I first began programming as a teenager, one of the mental hurdles I had to get over was asking the computer to "too much"; like, I would feel bad writing a nested loop --- that can't possibly be the right answer! What a chore for the computer! It didn't take me too long to figure out that was the whole point of computers. To me, it's the same thing with LLMs spinning on something. Who gives a shit? It's not me wasting that time.
  - kasey_junk 1y ago
    I’m now at the point where I tell it to do something _and then drive to Wisconsin_ giving feedback at the Marz cheese castle and continuing my trip.
    - tptacek 1y ago
      I feel like you might know where I'm coming from being confused at people's reaction to this stuff. This is science fiction. I think it's just not sinking in with people. If I could bet on this, I would bet everything I could on "skill with LLMs" being the high order bit of being an effective software developer 5 years from now.

viraptor 1y ago

Depends on the code, but often yes. The less you care about that specific result, the more efficient it is. One-off tools under 2k lines where you can easily verify the result? Why would I not generate that and save time for more interesting stuff?

layer8 1y ago
One-off tools seem to be more ops then dev, to me.
- jiggawatts 1y ago
  I’ve used Gemini Pro 2.5 to generate code syntax rewrite tools from me. I asked it to use a compiler SDK that I know from experience is fiddly and frustrating to use. It gave me a working tool in about 10 minutes, while I was actively listening to a meeting. Not all scripts are “ops”!
- viraptor 1y ago
  I haven't heard the "scripting is not programming" or similar take since newsgroups. It's really time to let it die.
  - layer8 1y ago
    That’s not what I wrote. It’s just that in my dev work I only rarely have the need for one-off tools.
    - sagarm 1y ago
      Do you not debug, optimize, analyze ...? Copilot is especially valuable for throwaway (or nearly throwaway) log parsing/analysis, for example.
      - layer8 1y ago
        I do, but I have all the software I need for that. I may use LLMs in that context, but for the actual analysis, not for generating one-off tools.

Aurornis 1y ago

Lately I look for opportunities to have the LLM do some easy small work while I go work on some harder task in parallel. I've also tried asking the LLM to come up with a proposed solution while I work on my own implementation at the same time. LLMs can also be much faster if a task requires some repetitive work. When I recognize a task like that, I try coding the first version and then ask the LLM to follow my pattern for the other areas where I need to repeat the work.

zackify 1y ago
Agree with each of these points so much! That’s why I really like copilot agent and codex right now. Even more parallel stuff and from my phone when I’m just thinking of ideas.
mwcampbell 1y ago
How did we end up here, just accepting that some coding tasks require repetitive work, and turning to a probabilistic text synthesizer that requires massive training data to automate that? We're having this discussion on a site whose founder famously wrote, over 20 years ago, that succinctness is power, and even wrote this site in a programming language that he designed to take that principle as far as he could. Now, two decades later, why have we so completely retreated from that dream? I bear some responsibility for this, since I was one of the people who basically said, in the 2010s, that we should just give up and use popular languages like JavaScript because they're popular. I regret that now.
- Aeolun 1y ago
  Some things are better with more code. Sets of data in particular (strongly typed). Sometimes you need to modify those sets of data and it doesn’t require enough work to write a whole script for, but you still don’t really want to spend the time manually modifying everything. LLM’s are really nice in those instances.
  - klabb3 1y ago
    Plain data in all glory but I’d rather (deterministically) generate and modify it than having big blobs checked in. If I have too much copy pasted data I often forget to modify until the test case runs and I realize it. This creates a false sense of confidence since there could be tests that pass and nobody ever checks they’re wrong. Essentially, minimize the number of human steps to verify it looks right which typically means optimizing for the least amount of code. Flashback to when I committed a suite of tests in Python that were indented one tab too much, resulting in them not running at all. This passed code review (on a FAANG company) and was discovered months later from an unrelated bug. The point is even unit tests have a very human element to them.

simonw 1y ago

Depends how good you are at reading and reviewing code. If you're already good at that LLMs are a huge productivity boost.

motorest 1y ago

> is it really more efficient to have an LLM generate code, then review that code, fix errors and spend some time to fully understand it? The answer is always "it depends". There are some drudge work tasks that are brilliantly done by LLMs, such as generating unit tests or documentation. Often the first attempt is not great, but iterating over them is so fast that you can regenerate everything from scratch a dozen times before you spend as much time as you would do if you wrote them yourself. It also depends on what scope you're working on. Small iterations have better results than grand redesigns. Context is also critical. if your codebase is neatly organized with squeaky clean code then LLMs generate better recommendations. If your codebase is a mess with inconsistent styles in spaghetti code then your prompts tend to generate more of the same.

max_on_hn 1y ago

(Disclaimer: I built and sell a product around that workflow) It often is, if you pick the right tasks (and more tasks fall into that bucket every few weeks). You can get a simple but fully-working app out of a single prompt, though quality varies widely unless you’re very specific. Once you have a codebase, agent output quality comes down to architecture and tests. If you have a scalable architecture with well-separated concerns, a solid integration test harness with examples, and good documentation (features, stack, procedures, design constraints), then getting the exact change you want is a matter of how well you can articulate what you want. One more asterisk, the development environment has to support the agent: like a human, agents work well with compiler feedback, and better with testing tools and documentation/internet access (yes my agents have these). I use CheepCode to work on itself, but I am still building up the test library and preview environments to de-risk merging non-trivial PRs that I haven’t pulled down and run locally. I also use it to work on other apps that I'm building, and since those are far more self-contained / easier to test, I get much better results there. If you want to put less effort into describing what you want, have a chat with an AI to generate tickets. Then paste those tickets into Linear and let CheepCode agents rip through them. I’ve got tooling in the works that will make that much easier, but I can only be in so many places at once as a bootstrapped founder :-)

teemur 1y ago

If you can identify blocks of code you need to write that are easy to define reasonably well, easy to review/verify that it is written correctly but still burdensome to actually write, LLMs are your new best friend. I don't know about how other people think/write, but I seem to have a lot of that kind of stuff on my table. The difficult part to outsource to LLMs is how to connect these easy blocks, but luckily thats the part I find fun in coding, not so much writing the boring simple stuff.

jsosshbfn 1y ago

LLMs tend to type a little faster than humans... so for straightforward code, yes?

mattlondon 1y ago

I frequently use it to assemble the boiler plate tedious stuff. E.g. add a new test file and wire up all the build rules etc, then implement tests x y z. Or the other day I had to create an input form in an Angular app and I just asked for the 7 or 8 fields I wanted and it did it and wired it all up for me (both template and Typescript). It can do those sorts of things in a few seconds and saves me maybe only 15-20 minutes but the mental relief of me not having to do it is great, even if the time saving is relatively low. The other thing is sometimes just writing out method signatures with all the right types and conversions/casts etc between types/interfaces/classes etc when I can't be bothered to do all the look ups myself ("Create a method here called foo that accepts a Bar instance and converts it to Baz. Return type should be Quux - add a conversion from Baz to a new Quux instance before the final return - use the builder pattern and map the constant magic-strings in blah.ts to appropriate enum values in Quux." Etc etc and then I write the logic in the middle) Again not a huge time saving but it mentally lightens the load and keeps you concentrating on the problem rather than the minutia

IshKebab 1y ago

In many cases yes. In many cases no. Overall, it can save you a lot of time even accounting for time wasted when you give it more than it can handle and you give up and do it all yourself.