4 ms·

> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes. I worked in life sciences for a bit.

> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes. I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircraft). It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place. (At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color) This is something most eng shops do not have the discipline for since it requires some diligence. Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.

drob 16d ago
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
skinfaxi 16d ago
> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered. I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
- jaggederest 16d ago
  I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence. I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
  - skinfaxi 16d ago
    I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
- DonCrespo 16d ago
  [flagged]
mlaretallack 16d ago
This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis. I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
grey-area 16d ago
The more important question - how would you actually get LLM agents to follow the instructions in your ever-growing CAPA reliably? It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
- CraigJPerry 16d ago
  Why couldn't you express all those as test cases rather than instructions? In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..." I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
  - palmotea 16d ago
    > Why couldn't you express all those as test cases rather than instructions? How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
  - CharlieDigital 16d ago
    You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases. The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures. In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
- jaggederest 16d ago
  You stop relying on the agents following instructions exactly. You need two pieces: a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it. b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift. The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
jeffnash 16d ago
Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks. It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree? This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
jayd16 16d ago
Honestly I feel like I'm taking crazy pills these days. Write a program! If it has a bug, debug it and update the program. Use an LLM to debug it if you must. If you're doing something repeatable, then use a traditional automation scheme. If an LLM has to handle a subsection of the program, so be it. Just wrap that part with clean inputs and outputs. Why does it feel like we've forgotten how to automate things?
- andrekandre 16d ago
  > Why does it feel like we've forgotten how to automate things? everybody is too busy everyday fighting fires to garden all those standard automations; its much "easier" to just throw it at an llm and move on (just saying what i see)
ericmcer 16d ago
That has been a part of post mortems everywhere I work, there was never a name for it though. Just someone would say: “how do we prevent this in the future?” which usually spawns a few tasks that the person who caused the issue or others will pick up.
nomel 16d ago
I don't see how this could work, right now, since every failing I've had was just generic stupidity of the AI, which is brilliant one minute, and a complete idiot the next. I think correction actions still require the ability to execute them, which (in all the cases I've had) would require more capable models. Long term, I think you're probably correct.
bryanrasmussen 16d ago
the history of assembly lines would be instructive. I don't think we have achieved a fully automated assembly line yet, but certainly they are much more automated than they were when Ford set up the first one.

drob 16d ago

Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.

skinfaxi 16d ago

> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered. I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.

jaggederest 16d ago
I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence. I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
- skinfaxi 16d ago
  I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
DonCrespo 16d ago
[flagged]

mlaretallack 16d ago

This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis. I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.

grey-area 16d ago

The more important question - how would you actually get LLM agents to follow the instructions in your ever-growing CAPA reliably? It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.

CraigJPerry 16d ago
Why couldn't you express all those as test cases rather than instructions? In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..." I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
- palmotea 16d ago
  > Why couldn't you express all those as test cases rather than instructions? How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
- CharlieDigital 16d ago
  You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases. The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures. In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
jaggederest 16d ago
You stop relying on the agents following instructions exactly. You need two pieces: a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it. b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift. The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.

jeffnash 16d ago

Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks. It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree? This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.

jayd16 16d ago

Honestly I feel like I'm taking crazy pills these days. Write a program! If it has a bug, debug it and update the program. Use an LLM to debug it if you must. If you're doing something repeatable, then use a traditional automation scheme. If an LLM has to handle a subsection of the program, so be it. Just wrap that part with clean inputs and outputs. Why does it feel like we've forgotten how to automate things?

andrekandre 16d ago
> Why does it feel like we've forgotten how to automate things? everybody is too busy everyday fighting fires to garden all those standard automations; its much "easier" to just throw it at an llm and move on (just saying what i see)

ericmcer 16d ago

That has been a part of post mortems everywhere I work, there was never a name for it though. Just someone would say: “how do we prevent this in the future?” which usually spawns a few tasks that the person who caused the issue or others will pick up.

nomel 16d ago

I don't see how this could work, right now, since every failing I've had was just generic stupidity of the AI, which is brilliant one minute, and a complete idiot the next. I think correction actions still require the ability to execute them, which (in all the cases I've had) would require more capable models. Long term, I think you're probably correct.

bryanrasmussen 16d ago

the history of assembly lines would be instructive. I don't think we have achieved a fully automated assembly line yet, but certainly they are much more automated than they were when Ford set up the first one.