57 ms·

Perhaps I've missed a few weeks worth of progress, but I don't think that AIs have become more trustworthy, the errors are just more subtle. If the code doesn'

Perhaps I've missed a few weeks worth of progress, but I don't think that AIs have become more trustworthy, the errors are just more subtle. If the code doesn't compile, that's easy to spot. If the code compiles but doesn't work, that's still somewhat easy to spot. If the code compiles and works, but it does the wrong thing in some edge case, or has a security vulnerability, or introduces tech debt or dubious architectural decisions, that's harder to spot but doesn't reduce the review burden whatsoever. If anything, "truthy" code is more mentally taxing to review than just obviously bad code.

christoff12 6mo ago
This has generally been the case, though. As mentioned in the post, "You want solutions that are proven to work before you take a risk on them" remains true and will be place where the edges are found.
- zarzavat 6mo ago
  It's about responsibility. If I get pwned because my AI agent wrote code that had a security vulnerability, none of my users are going to accept the excuse that I used AI and it's a brave new world. I will get the blame, not Anthropic or OpenAI or Google but me. The same goes for if my AI generated code leads to data loss, or downtime, or if uses too many resources, or it doesn't scale, or it gives out error messages like candy. The buck stops with me and therefore I have to read the code, line-by-line, carefully. It's not even a formality. I constantly find issues with AI generated code. These things are lazy and often just stub out code instead of making a sober determination of whether the functionality can be stubbed out or not. You could say "just AI harder and get the AI to do the review", and I do this a lot, but reviewing is not a neutral activity. A review itself can be harmful if it flags spurious issues where the fix creates new problems. So I still have to go through the AI generated review issue-by-issue and weed out any harmful criticism.
  - user34283 6mo ago
    On the other hand, I don’t need to review carefully every line of code in my thumbnail generator and associated UI. My nonexistent backend isn’t going to be pwned if there is a bug in the thumbnail generation. After the QA testing on my device, a quick scroll through of the code is enough. Maybe prompt „are errors during thumbnail generation caught to prevent app crashes?“ if we‘re feeling extra cautious today. And just like that it saved a day of work.
    - jaggederest 6mo ago
      > My nonexistent backend isn’t going to be pwned if there is a bug in the thumbnail generation. Hmm. Historically image editing was one of the easier to exploit security holes in many systems. How do you feel about having unknown entities having shell inside your datacenter or vpc?
      - user34283 6mo ago
        I feel pretty good about the odds of attackers exploiting security holes in image editing functions my app does not have, in order to enter my also nonexistent datacenter or vpc.
    - djhn 6mo ago
      But a thumbnail generator is a 1 hour task at best if you’re on a solo greenfield project and it’ll still be a 6 week project at an enterprise, even with AI.
      - user34283 6mo ago
        I would be impressed if you implement it in an hour with the following features: - webview fallback with canvas capture for codecs not supported in the default player - detecting blank frames and diff between thumbnails to maximize variety - UI integration to visualize progress and pending thumbnails, batched updates to the gallery - versioning scheme and backfill for missing/outdated thumbnail formats Honestly, a day seems rather optimistic to me. Maybe if I was an expert for this platform and would have implemented a similar feature before, then I could hope to do it in a day. If I had to handwrite it and estimate it for Scrum at work, I‘d budget a week.
  - jaggederest 6mo ago
    I think there's a couple levels here: First of all, building a system that constrains the output of the AI sufficiently, whether that's typing, testing, external validation, or manual human review in extremis. That gets you the best result out of whatever harness or orchestration you're using. Secondly, there's the level at which you're intervening, something along the hierarchy of "validate only usage from the customer perspective" to "review, edit, and validate every jot and tiddle of the codebase and environment". I think for relatively low importance things reviewing at the feature level (all code, but not interim diffs) is fine, but if you're doing network protocol you better at least validate everything carefully with fuzzing and prop testing or something like that. And then you've got how you structure your feedback to the LLM itself - is it an in-the-loop chat process, an edit-and-retry spec loop, go-nogo on a feature branch, or what? How does the process improve itself, basically? I agree with you entirely that the responsibility rests on the human, but there are a variety of ways to use these things that can increase or decrease the quality of code to time spent reviewing, and obviously different tasks have different levels of review scrutiny, as well.
xantronix 6mo ago
I know there are good uses of LLMs out there. I do. But. The current fever pitch mandates from above seem to want it applied liberally, and pushing back against that is so discouraging and often career-limiting as to wear the fabric of one's psyche threadbare. With all the obvious problems being pointed out to people, there are just as many workarounds; and these workarounds, as is often revealed shortly thereafter, have their own problems, which beget new solutions, ad infinitum. At some point it genuinely seems like all this work is for the sake of the machine itself. I suppose that is true: The real goal has become obscured at so many firms today, that all that remains is the LLM. Are the people betting the farm and helping implement the visions of those who have done so guaranteed a soft exit to cushion them from the consequences, or is rationality really being discarded altogether? Sure, sound engineering principles can help work around these problems, but what efficiency is truly gained, in terms of cognitive load, developer time, money, or finite resources? Or were those ever an earnest concern?
- Daishiman 6mo ago
  There's two sides to the AI mandates. The degenerate side is clueless upper management and fad-driven engineering. We have talked extensively about this. There is a more rational side to it that I've seen in my org: some engineers absolutely refuse to use AI and as a consequence they are now, clearly and objectively, much less productive than other engineers. The thing is, you still need to learn how to use the tool, so a nontrivial percentage of obstinate engineers need to be driven to use this in the same way that some developers have refused to use Docker or k8s or whatever.
  - callc 6mo ago
    Ah yes, we must force these obstinate engineers to the right path! Only after getting everyone to see the light will they understand and thank us for boundless productivity!! /s Perhaps these “obstinate” engineers have good reason in their decision. And it should be their decision! To be so confident in what is “the right way (TM)” and try to force it onto others is... revealing.
    - 6mo ago
asdfman123 6mo ago
You can direct LLMs to do test-driven development, though. Write several tests, then make sure the code matches it. And also make sure the agent organizes the code correctly.
- CharlieDigital 6mo ago
  The LLM obliges and writes a lot of useless tests. I have asked devs to delete several tests in the last day alone.
  - seanw444 6mo ago
    "I don't trust this giant statistical model to generate correct code, so to fix it, I'm going to have this giant statistical model generate more code to confirm that the other code it generated is correct." I swear I'm living through mass hysteria.
    - hyperadvanced 6mo ago
      A lot of times the act of specifying test criteria prevents developers from accidentally vibe coding themselves into a bad implementation. You can then read the tests and verify that it does what you want it to. You can read the code! I’m not saying that it’s all hunky dory, but you use AI for straight up test driven development to catch edge cases and correct sloppy implementations before they even get coded by your giant chaos machine.
  - asdfman123 6mo ago
    Well, yeah, you don't just make it bang out a bunch of useless code without monitoring it. You instruct it to write the code you want to be written. You still have to know how to develop, it just makes you faster.
hintymad 6mo ago
> I don't think that AIs have become more trustworthy, the errors are just more subtle. Honest question: what about the counter-argument that humans make subtle mistakes all the time, so why do we treat AI any differently? A difference to me is that when we manually write code, we reason about the code carefully with a purpose. Yes we do make mistakes, but the mistakes are grounded in a certain range. In contrast, AI generated code creates errors that do not follow common sense. That said, I don't feel this differentiation is strong enough, and I don't have data to back it up.
- sumeno 6mo ago
  Humans can't make mistakes at the sheer scale that AI can. Yes, as an engineer I make mistakes, but I could never make as many mistakes per day as an LLM can
  - deleted 6mo ago
    [deleted]
  - throwuxiytayq 6mo ago
    Obviously, the measure isn’t mistakes per day, it’s mistakes per LOC. And that’s not the whole story either - AI self-corrects in addition to being corrected by the operator. If the operator’s committed bugs/LOC rate is as low as the unaugmented programmer’s bugs/LOC, you always choose the AI operator. If it’s higher, it might still be viable to choose them if you care about velocity more than correctness. I’m a slow, methodical programmer myself, but it’s not clear to me that I have a moat.
- BoorishBears 6mo ago
  This is like having a coworker who's as skilled as you if not more skilled, but also an alien. Their mental model doesn't map cleanly enough to yours, and so where for a human you'd have some way to follow their thought patterns and identify mistakes, here the alien makes mistakes that don't add up. Like the alien has encyclopedic knowledge of op codes in some esoteric soviet MCU but sometimes forgets how to look for a function definition, says "It looks like the read tool failed, that's ok, I can just make a mock implementation and comment out the test for now."
sanderjd 6mo ago
Yeah I relate to this. I think working in smaller chunks helps a lot. (Just like how it is for work done by humans!)
esalman 6mo ago
This. My manager reported couple of days ago that copilot manipulated some tests in order to make edge cases pass. We have standalone prototypes for our product, so it was easy to catch, but actually going in to debug and fix was much harder than expected. It absolutely did nothing to increase confidence on copilot though. I personally manually accept each line of code copilot writes, unless it's a skill/mcp server we have no plan to deploy.

christoff12 6mo ago

This has generally been the case, though. As mentioned in the post, "You want solutions that are proven to work before you take a risk on them" remains true and will be place where the edges are found.

zarzavat 6mo ago
It's about responsibility. If I get pwned because my AI agent wrote code that had a security vulnerability, none of my users are going to accept the excuse that I used AI and it's a brave new world. I will get the blame, not Anthropic or OpenAI or Google but me. The same goes for if my AI generated code leads to data loss, or downtime, or if uses too many resources, or it doesn't scale, or it gives out error messages like candy. The buck stops with me and therefore I have to read the code, line-by-line, carefully. It's not even a formality. I constantly find issues with AI generated code. These things are lazy and often just stub out code instead of making a sober determination of whether the functionality can be stubbed out or not. You could say "just AI harder and get the AI to do the review", and I do this a lot, but reviewing is not a neutral activity. A review itself can be harmful if it flags spurious issues where the fix creates new problems. So I still have to go through the AI generated review issue-by-issue and weed out any harmful criticism.
- user34283 6mo ago
  On the other hand, I don’t need to review carefully every line of code in my thumbnail generator and associated UI. My nonexistent backend isn’t going to be pwned if there is a bug in the thumbnail generation. After the QA testing on my device, a quick scroll through of the code is enough. Maybe prompt „are errors during thumbnail generation caught to prevent app crashes?“ if we‘re feeling extra cautious today. And just like that it saved a day of work.
  - jaggederest 6mo ago
    > My nonexistent backend isn’t going to be pwned if there is a bug in the thumbnail generation. Hmm. Historically image editing was one of the easier to exploit security holes in many systems. How do you feel about having unknown entities having shell inside your datacenter or vpc?
    - user34283 6mo ago
      I feel pretty good about the odds of attackers exploiting security holes in image editing functions my app does not have, in order to enter my also nonexistent datacenter or vpc.
  - djhn 6mo ago
    But a thumbnail generator is a 1 hour task at best if you’re on a solo greenfield project and it’ll still be a 6 week project at an enterprise, even with AI.
    - user34283 6mo ago
      I would be impressed if you implement it in an hour with the following features: - webview fallback with canvas capture for codecs not supported in the default player - detecting blank frames and diff between thumbnails to maximize variety - UI integration to visualize progress and pending thumbnails, batched updates to the gallery - versioning scheme and backfill for missing/outdated thumbnail formats Honestly, a day seems rather optimistic to me. Maybe if I was an expert for this platform and would have implemented a similar feature before, then I could hope to do it in a day. If I had to handwrite it and estimate it for Scrum at work, I‘d budget a week.
- jaggederest 6mo ago
  I think there's a couple levels here: First of all, building a system that constrains the output of the AI sufficiently, whether that's typing, testing, external validation, or manual human review in extremis. That gets you the best result out of whatever harness or orchestration you're using. Secondly, there's the level at which you're intervening, something along the hierarchy of "validate only usage from the customer perspective" to "review, edit, and validate every jot and tiddle of the codebase and environment". I think for relatively low importance things reviewing at the feature level (all code, but not interim diffs) is fine, but if you're doing network protocol you better at least validate everything carefully with fuzzing and prop testing or something like that. And then you've got how you structure your feedback to the LLM itself - is it an in-the-loop chat process, an edit-and-retry spec loop, go-nogo on a feature branch, or what? How does the process improve itself, basically? I agree with you entirely that the responsibility rests on the human, but there are a variety of ways to use these things that can increase or decrease the quality of code to time spent reviewing, and obviously different tasks have different levels of review scrutiny, as well.

xantronix 6mo ago

I know there are good uses of LLMs out there. I do. But. The current fever pitch mandates from above seem to want it applied liberally, and pushing back against that is so discouraging and often career-limiting as to wear the fabric of one's psyche threadbare. With all the obvious problems being pointed out to people, there are just as many workarounds; and these workarounds, as is often revealed shortly thereafter, have their own problems, which beget new solutions, ad infinitum. At some point it genuinely seems like all this work is for the sake of the machine itself. I suppose that is true: The real goal has become obscured at so many firms today, that all that remains is the LLM. Are the people betting the farm and helping implement the visions of those who have done so guaranteed a soft exit to cushion them from the consequences, or is rationality really being discarded altogether? Sure, sound engineering principles can help work around these problems, but what efficiency is truly gained, in terms of cognitive load, developer time, money, or finite resources? Or were those ever an earnest concern?

Daishiman 6mo ago
There's two sides to the AI mandates. The degenerate side is clueless upper management and fad-driven engineering. We have talked extensively about this. There is a more rational side to it that I've seen in my org: some engineers absolutely refuse to use AI and as a consequence they are now, clearly and objectively, much less productive than other engineers. The thing is, you still need to learn how to use the tool, so a nontrivial percentage of obstinate engineers need to be driven to use this in the same way that some developers have refused to use Docker or k8s or whatever.
- callc 6mo ago
  Ah yes, we must force these obstinate engineers to the right path! Only after getting everyone to see the light will they understand and thank us for boundless productivity!! /s Perhaps these “obstinate” engineers have good reason in their decision. And it should be their decision! To be so confident in what is “the right way (TM)” and try to force it onto others is... revealing.
  - 6mo ago

asdfman123 6mo ago

You can direct LLMs to do test-driven development, though. Write several tests, then make sure the code matches it. And also make sure the agent organizes the code correctly.

CharlieDigital 6mo ago
The LLM obliges and writes a lot of useless tests. I have asked devs to delete several tests in the last day alone.
- seanw444 6mo ago
  "I don't trust this giant statistical model to generate correct code, so to fix it, I'm going to have this giant statistical model generate more code to confirm that the other code it generated is correct." I swear I'm living through mass hysteria.
  - hyperadvanced 6mo ago
    A lot of times the act of specifying test criteria prevents developers from accidentally vibe coding themselves into a bad implementation. You can then read the tests and verify that it does what you want it to. You can read the code! I’m not saying that it’s all hunky dory, but you use AI for straight up test driven development to catch edge cases and correct sloppy implementations before they even get coded by your giant chaos machine.
- asdfman123 6mo ago
  Well, yeah, you don't just make it bang out a bunch of useless code without monitoring it. You instruct it to write the code you want to be written. You still have to know how to develop, it just makes you faster.

hintymad 6mo ago

> I don't think that AIs have become more trustworthy, the errors are just more subtle. Honest question: what about the counter-argument that humans make subtle mistakes all the time, so why do we treat AI any differently? A difference to me is that when we manually write code, we reason about the code carefully with a purpose. Yes we do make mistakes, but the mistakes are grounded in a certain range. In contrast, AI generated code creates errors that do not follow common sense. That said, I don't feel this differentiation is strong enough, and I don't have data to back it up.

sumeno 6mo ago
Humans can't make mistakes at the sheer scale that AI can. Yes, as an engineer I make mistakes, but I could never make as many mistakes per day as an LLM can
- deleted 6mo ago
  [deleted]
- throwuxiytayq 6mo ago
  Obviously, the measure isn’t mistakes per day, it’s mistakes per LOC. And that’s not the whole story either - AI self-corrects in addition to being corrected by the operator. If the operator’s committed bugs/LOC rate is as low as the unaugmented programmer’s bugs/LOC, you always choose the AI operator. If it’s higher, it might still be viable to choose them if you care about velocity more than correctness. I’m a slow, methodical programmer myself, but it’s not clear to me that I have a moat.
BoorishBears 6mo ago
This is like having a coworker who's as skilled as you if not more skilled, but also an alien. Their mental model doesn't map cleanly enough to yours, and so where for a human you'd have some way to follow their thought patterns and identify mistakes, here the alien makes mistakes that don't add up. Like the alien has encyclopedic knowledge of op codes in some esoteric soviet MCU but sometimes forgets how to look for a function definition, says "It looks like the read tool failed, that's ok, I can just make a mock implementation and comment out the test for now."

sanderjd 6mo ago

Yeah I relate to this. I think working in smaller chunks helps a lot. (Just like how it is for work done by humans!)

esalman 6mo ago

This. My manager reported couple of days ago that copilot manipulated some tests in order to make edge cases pass. We have standalone prototypes for our product, so it was easy to catch, but actually going in to debug and fix was much harder than expected. It absolutely did nothing to increase confidence on copilot though. I personally manually accept each line of code copilot writes, unless it's a skill/mcp server we have no plan to deploy.