15 ms·

The Hallucination Defense

56 points by niyikiza 9mo ago

JohnFen 9mo ago
> “The AI hallucinated. I never asked it to do that.” > That’s the defense. And here’s the problem: it’s often hard to refute with confidence. Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
- salawat 9mo ago
  Except for the fact that that very accountability sink is relied on by senior management/CxO's the world over. The only difference is that before AI, it was the middle manager's fault. We didn't tell anyone to break the law. We just put in place incentive structures that require it, and play coy, then let anticipatory obedience do the rest. Bingo. Accountability severed. You can't prove I said it in a court of law, and skeevy shit gets done because some poor bloke down the ladder is afraid of getting fired if he doesn't pull out all the stops to meet productivity quotas. AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law.
  - pixl97 9mo ago
    Hence why many life and death things require licencing and compliance, and tend to come with very long paper trails. The software world has been very allergic to getting anywhere near the vicinity of a system like that.
    - salawat 9mo ago
      Did I give the impression that the phenomena was unique to software? Hell, Boeing was a shining example of the principle in action with 737 MAX. Don't get much more "people live and die by us, and we know it (but management set up the culture and incentives to make a deathtrap anyway)." No one to blame of course. These things just happen. Licensure alone doesn't solve all these ills. And for that matter, once regulatory capture happens, it has a tendency to make things worse due to consolidation pressure.
  - Muromec 9mo ago
    >AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law. AI is worse in that regard, because, although you can't explain why it does so, you can point a finger at it, say "we told you so" and provide the receipts of repeated warnings that the thing has a tendency of doing the things.
- niyikiza 9mo ago
  You're right, they should be responsible. The problem is proving it. "I asked it to summarize reports, it decided to email the competitor on its own" is hard to refute with current architectures. And when sub-agents or third-party tools are involved, liability gets even murkier. Who's accountable when the action executed three hops away from the human? The article argues for receipts that make "I didn't authorize that" a verifiable claim
  - bulatb 9mo ago
    There's nothing to prove. Responsibility means you accept the consequences for its actions, whatever they are. You own the benefit? You own the risk. If you don't want to be responsible for what a tool that might do anything at all might do, don't use the tool. The other option is admitting that you don't accept responsibility, not looking for a way to be "responsible" but not accountable.
    - tossandthrow 9mo ago
      Sounds good in theory, doesn't work in reality. Had it worked then we would have seen many more CEOs in prison.
      - NoMoreNicksLeft 9mo ago
        The veil of liability is built into statute, and it's no accident. Such so magic forcefield exists for you, though.
      - walt_grata 9mo ago
        There being a few edge cases where it doesn't work in doesn't mean it doesn't work in the majority of cases and that we shouldn't try to fix the edge cases.
      - bulatb 9mo ago
        We're taking about different things. To take responsibility is volunteering to accept accountability without a fight. In practice, almost everyone is held potentially or actually accountable for things they never had a choice in. Some are never held accountable for things they freely choose, because they have some way to dodge accountability. The CEOs who don't accept accountability were lying when they said they were responsible.
- doctorpangloss 9mo ago
  Wait till you find out about “pedal confusion.”
- nerdsniper 9mo ago
  The distinction some people are making is between copy/pasting text vs agentic action. Generally mistakes "work product" as in output from ChatGPT that the human then files with a court, etc. are not forgiven, because if you signed the document, you own its content. Versus some vendor-provided AI Agent which simply takes action on its own that a "reasonable person" would not have expected it to. Often we forgive those kinds of software bloopers.
  - ori_b 9mo ago
    If you put a brick on the accelerator of a car and hop out, you don't get to say "I wasn't even in the car when it hit the pedestrian".
    - Shalomboy 9mo ago
      This is true for bricks, but it is not true if your dog starts up your car and hits a pedestrian. Collisions caused by non-human drivers are a fascinating edge case for the times we're in.
      - victorbjorklund 9mo ago
        I don’t know where you from but at least in Sweden you have strict liability for anything your dog does
      - deleted 9mo ago
        [deleted]
      - freejazz 9mo ago
        Prima facie negligence = liability
      - ori_b 9mo ago
        In the USA, at least, it seems pet owners are liable for any harm their pets do.
      - cess11 9mo ago
        Legally, in a lot of jurisdictions, a dog is just your property. What it does, you did, usually with presumed intent or strict liability.
- ibejoeb 9mo ago
  Yeah. Legal will need to catch up to deal with some things, surely, but the basic principles for this particular scenario aren't that novel. If you're a professional and have an employee acting under your license, there's already liability. There is no warrant concept (not that I can think of right now, at least) that will obviate the need to check the work and carry professional liability insurance. There will always be negligence and bad actors. The new and interesting part is that while we have incentives and deterrents to keep our human agents doing the right thing, there isn't really an analog to check the non-human agent. We don't have robot prison yet.
- imiric 9mo ago
  That's quickly becoming difficult to determine. The workflow of starting dozens or hundreds of "agents" that work autonomously is starting to gain traction. The goal of people who work like this is to completely automate software development. At some point they want to be able to give the tool an arbitrary task, presumably one that benefits them in some way, and have it build, deploy, and use software to complete it. When millions of people are doing this, and the layers of indirection grow in complexity, how do you trace the result back to a human? Can we say that a human was really responsible for it? Maybe this seems simple today, but the challenges this technology forces on society are numerous, and we're far from ready for it.
  - niyikiza 9mo ago
    This is the problem we're working on. When orchestrators spawn sub-agents spawn tools, there's no artifact showing how authority flowed through the chain. Warrants are a primitive for this: signed authorization that attenuates at each hop. Each delegation is signed, scope can only narrow, and the full chain is verifiable at the end. Doesn't matter how many layers deep.
  - Wobbles42 9mo ago
    Translation: People want to use a tool and not be liable for the result. People not wanting to be liable for their actions is not new. AI hasn't changed anything here, it's just a new lame excuse.
- NedF 9mo ago
  [dead]
- godelski 9mo ago
  > It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not. I hate to ask, but did you RTFA? Scrolling down ever so slightly (emphasis not my own) | *Who authorized this class of action, for which agent identity, under what constraints, for how long; and how did that authority flow?* | A common failure mode in agent incidents is not “we don’t know what happened,” but: | > We can’t produce a crisp artifact showing that a specific human explicitly authorized the scope that made this action possible. They explicitly state that the problem is you don't know which human to point at.
  - plastic041 9mo ago
    > They explicitly state that the problem is you don't know which human to point at. The point is "explicitly authorized", as the article emphasizes. It's easy to find who ran the agent(article assumes they have OAuth log). This article is about 'Everyone knows who did it, but did they do it on purpose? Our system can figure it out'
- ragBagger 9mo ago
  I feel like you have missed the point of this. It isn't to completely absolve the user of liability, it's to prove malice instead of incompetence. If the user claims that they only authorized the bot to review files, but they've warranted the bot to both scan every file and also send emails to outside sources, the competitors in this case, then you now have proof that the user was planning on committing corporate espionage. To use a more sane version of an example below, if your dog runs outside the house and mauls a child, you are obviously guilty of negligence, but if there's proof of you unleashing the dog and ordering the attack, you're guilty of murder.
RobotToaster 9mo ago
If an employee does something during his employment, even if he wasn't told directly to do it, the company can be held vicariously liable, how is this any different?
- apercu 9mo ago
  I agree with you but you can’t jail a gen-ai model, I guess, is where the difference lies?
  - phailhaus 9mo ago
    Nobody tries to jail Microsoft Word, they jail the person using it.
    - gorjusborg 9mo ago
      Nobody tries to jail the automobile being driven when it hits a pedestrian when on cruise control. The driver is responsible for knowing the limits of the tool and adjusting accordingly.
  - LeifCarrotson 9mo ago
    "The company can be held vicariously liable" means that in this analogy, the company represents the human who used AI inappropriately, and the employee represents the AI model that did something it wasn't directly told to do.
tboyd47 9mo ago
Anytime someone gives you unverified information, they're asking you to become their guinea pig.
noitpmeder 9mo ago
This is some absolute BS. In the current day and age you are 1000% responsible for the externalities of your use of AI. Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider. Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users. Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!! And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business
- ceejayoz 9mo ago
  T&Cs aren't ironclad. One in which you sell yourself into slavery, for example, would be illegal in the US. All those "we take no responsibility for the [valet parking|rocks falling off our truck|exploding bottles]" disclaimers are largely attempts to dissuade people from trying. As an example, NY bans liability waivers at paid pools, gyms, etc. The gym will still have you sign one! But they have no enforcement teeth beyond people assuming they're valid. https://codes.findlaw.com/ny/general-obligations-law/gob-sect-5-326/ https://codes.findlaw.com/ny/general-obligations-law/gob-sec...
  - noitpmeder 9mo ago
    So I can pass on contact breaches due to bugs in software I maintain due to hallucinations by the AI that I used to write the software?? Absolutely no way. "But the AI wrote the bug." Who cares? It could be you, your relative, your boss, your underling, your counterpart in India, ... Your company provided some reasonable guarantee of service (whether explitly enumerated in a contact or not) and you cannot just blindly pass the buck. Sure, after you've settled your claim with the user, maybe TRY to go after the upstream provider, but good luck. (Extreme example) -- If your company produces a pacemaker dependent on AWS/GCP/... and everyone dies as soon as cloudflare has a routing outage that cascades to your provider, oh boy YOU are fucked, not cloudflare or your hosting provider.
freejazz 9mo ago
What a stupid article from someone that has no idea when liability attaches. It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.
gamblor956 9mo ago
It's not a legal defense at all. Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses. However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output. And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).
rpodraza 9mo ago
What problem is this guy trying to solve? Sorry, but in the end, someone's gonna have to be responsible and it's not gonna be a computer program. Someone approved the program's use, it's no different to any other software. If you know agent can make mistakes then you need to verify everything manually, simple as.
- pixl97 9mo ago
  While we're a long way off from the day science fiction becomes fact, the world is going to shit itself if a self actionable AI bootstraps and causes havoc.
0xTJ 9mo ago
Why would that be any better of a defense than "that preschooler said that I should do it"? People are responsible for their work.
andrewflnr 9mo ago
How does the old proverb go? > A computer must never make a management decision, because a computer cannot be held accountable.
bitwize 9mo ago
aka the Shaggy Defense for the 2020s.
kazinator 9mo ago
This article is well-written insanity. With no amount of detailed logging makes "the AI did it" a valid excuse. It's just a tool. It's like blaming a loose bolt in a Boeing 737 on "screwdriver did it".
- kulahan 9mo ago
  If you tighten a bolt using a breaker bar and it fails unexpectedly, it really is the screwdriver’s fault though. Nobody can know if the bolt truly has 70 pounds of torque applied. Maybe it broke at 62, and will fail in 3 flights.
stronglikedan 9mo ago
If one of my reports came to me with that defense, I'd write them up twice. Once for whatever they did wrong, and once for insulting my intelligence and wasting my time with that "defense". On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.
thedudeabides5 9mo ago
a machine can never be held accountable but the person who turned it on can simple as
onoesworkacct 9mo ago
IMO everyone is missing the point of this thing. It's not an auth system or security boundary, it doesn't provide any security guarantees whatsoever, it doesn't do anything. The entire point is to cover a company's derriere should their agentic security apparatus (or lack thereof) fail to prevent malicious prompt injection etc. This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken
deleted 9mo ago
[deleted]
amelius 9mo ago
We should demand money back if an LLM hallucinates. And they should be liable.
- jbsimpson 9mo ago
  It's a fundamental limitations of LLMS - don't use them if you're worried about hallucinations.
Wobbles42 9mo ago
I don't understand what has changed here. AI is just branding. At the end of the day it's still just people using computer software to do stuff. There is going to be a person who did a thing at the end of the day -- either whoever wrote the software or whoever used the tool. The fact that software inexplicably got unreliable when we started stamping "AI" on the box shouldn't really change anything.
booleandilemma 9mo ago
It's the same thing as saying "It's not my fault, it's a bug in the spreadsheet". The human is ultimately responsible. Heck, it's the same thing as someone saying "the computer made a mistake". Computers don't make mistakes. People do. The hallucination defense is not a defense at all and this is a non-issue. There's nothing to talk about here. And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.
- Wobbles42 9mo ago
  If the LLMs don't start producing positive revenue soon, the consequences for society will be rather short term.
jbsimpson 9mo ago
Anyone using AI tools should assume that at least 20% of what it produces will be wrong: Missing details, inventing things, basic mistakes. Use AI if being 80% right quickly is fine. Otherwise if you have to do the analysis anyway because accuracy is critical, there's little point to the AI - its too unreliable.
overgard 9mo ago
I'm not sure I understand the point of this. If mistakes (or lies) aren't tolerable in the output, then whoever ran the agent should be responsible for reviewing the output and the resulting actions. I don't see how LLMs can be "responsible" for anything because they don't think in the way we do nor do they have motives in the way we do. The "Hallucination Defense" isn't a defense because end of the day, if you ran it, you're responsible, IMO.
Terr_ 9mo ago
A useful phrase for this when it's an intentional/permitted design: "Accountability Sinks." Or perhaps scapegoating at scale. https://news.ycombinator.com/item?id=43877301 https://news.ycombinator.com/item?id=43877301 - 398 comments https://news.ycombinator.com/item?id=41891694 https://news.ycombinator.com/item?id=41891694 - 308 comments
phs318u 9mo ago
Two things: 1. I vaguely recall that in the early days of Windows, the TOS explicitly told users they were not to use it for certain use-cases and the wording was something like "we don't warranty windows to be good for anything, but we EXPLICITLY do not want you to use Windows to do nuclear foo". I expect that if the big LLM vendors aren't already doing this, they soon will. Kind of like how Fox News claims that some of heads on the screen are not journalists at all but merely entertainers (and you wouldn't take anything they say seriously ergo we're not responsible for stuff that happens as a result of people listening to our heads on the screen). 2. IANAL but I believe that in most legal systems, responsibility requires agency. Leaving aside that we call these things "agents" (which is another thing I suspect will change in the marketing), they do not have agency. As a result they must be considered tools. Tools can be used according to the instructions/TOS, or not. If not - whatever it does is down to you. If used within guidelines - it's the manufacturer. So my conclusion is that the vendors - who have made EPIC bets on getting this tech into the hands of as many folks as possible and making it useful for pretty much anything you can think of - will be faced with a dilemma. At the moment, it seems like they believe that (just like the infamous Ford bean counters) the benefits of not restricting the TOS will far outweigh any consequences from bad things happening. Remains to be seen.
plastic041 9mo ago
This is an advertisement for a 'tenuo warrant'. So, I read its document[0]. Put simply, it works like this: 1. A person orders an AI agent to do A. 2. The agent issues a tenuo warrant for doing A. 3. The agent can now only use the tool to perform A. The article is about that 'warrant' can now be used in case of an incident because it contains information such as 'who ordered the task' and 'what authority was given'. I get the idea. This isn't about whether a person is responsible or not(because of course they are). It's more about whether it was intentional. However... wouldn't it be much easier to just save the prompt log? This article is based entirely on "But the prompt history? Deleted."(from the article) situation. [0]: https://tenuo.dev/concepts https://tenuo.dev/concepts
- niyikiza 9mo ago
  You've got the model right. And saving prompt logs does help with reconstruction. But warrants aren't just "more audit data." They're an authorization primitive enforced in the critical path: scope and constraints are checked mechanically before the action executes. The receipt is a byproduct. Prompt logs tell you what the model claimed it was doing. A warrant is what the human actually authorized, bound to an agent key, verifiable without trusting the agent runtime. This matters more in multi-agent systems. When Agent A delegates to Agent B, which calls a tool, you want to be able to link that action back to the human who started it. Warrants chain cryptographically. Each hop signs and attenuates. The authorization provenance is in the artifact itself.
  - plastic041 9mo ago
    But the AI agent still needs to determine which tool is necessary to mint the warrant. What happens if the agent makes a mistake when making warrant?
    - niyikiza 9mo ago
      A worker agent doesn't mint warrants. It receives them. Either it requests a capability and an issuer approves, or the issuer pushes a scoped warrant when assigning a task. Either way, the issuer signs and the agent can only act within those bounds. At execution time, the "verifier" checks the warrant: valid signatures, attenuation (scope only narrows through delegation), TTL (authority is task-scoped), and that the action fits the constraints. Only then does the call proceed. This is sometimes called the P/Q model: the non-deterministic layer proposes, the deterministic layer decides. The agent can ask for anything. It only gets what's explicitly granted. If the agent asks for the wrong thing, it fails closed. If an overly broad scope is approved, the receipt makes that approval explicit and reviewable.
dietr1ch 9mo ago
Blaming AI and expecting it to cover your ass is like saying that you wrote the code with your elbows and expecting that to make things OK.
michaelmrose 9mo ago
Shouldn't all agentic actions with meaningful outputs of importance like moving $48,000 simply be required to terminate in a human designed or verified output with a human in the loop attestation. Eg a list of transactions that isn't AI generated where the only actions that actually move money must operate on the data displayed in the human designed page. A human looks at this and says yes that is acceptable and becomes reasonable for that action.
- krapp 9mo ago
  The entire goal of AI is to not have humans in the loop at all. So while that should happen, it won't. They'll just add an extra layer of AI to do the verification.
  - michaelmrose 9mo ago
    If AI were good enough to detect hallucinations wouldn't that be built into the AI already?
    - niyikiza 9mo ago
      Exactly ... and that's why I'm skeptical of "AI verifies AI" as the primary safety mechanism. The verifier for moving money should be deterministic: constraints, allowlists, spend limits, invoice/PO matching, etc. The LLM can propose actions, but the execution should be gated by a human/polic-issued scope that's mechanically enforced. That's the whole point: constrain the non-deterministic layer with a deterministic one. [0] [0] https://tenuo.dev/constraints https://tenuo.dev/constraints

56 points by niyikiza 9mo ago

> “The AI hallucinated. I never asked it to do that.” > That’s the defense. And here’s the problem: it’s often hard to refute with confidence. Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.

salawat 9mo ago
Except for the fact that that very accountability sink is relied on by senior management/CxO's the world over. The only difference is that before AI, it was the middle manager's fault. We didn't tell anyone to break the law. We just put in place incentive structures that require it, and play coy, then let anticipatory obedience do the rest. Bingo. Accountability severed. You can't prove I said it in a court of law, and skeevy shit gets done because some poor bloke down the ladder is afraid of getting fired if he doesn't pull out all the stops to meet productivity quotas. AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law.
- pixl97 9mo ago
  Hence why many life and death things require licencing and compliance, and tend to come with very long paper trails. The software world has been very allergic to getting anywhere near the vicinity of a system like that.
  - salawat 9mo ago
    Did I give the impression that the phenomena was unique to software? Hell, Boeing was a shining example of the principle in action with 737 MAX. Don't get much more "people live and die by us, and we know it (but management set up the culture and incentives to make a deathtrap anyway)." No one to blame of course. These things just happen. Licensure alone doesn't solve all these ills. And for that matter, once regulatory capture happens, it has a tendency to make things worse due to consolidation pressure.
- Muromec 9mo ago
  >AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law. AI is worse in that regard, because, although you can't explain why it does so, you can point a finger at it, say "we told you so" and provide the receipts of repeated warnings that the thing has a tendency of doing the things.
niyikiza 9mo ago
You're right, they should be responsible. The problem is proving it. "I asked it to summarize reports, it decided to email the competitor on its own" is hard to refute with current architectures. And when sub-agents or third-party tools are involved, liability gets even murkier. Who's accountable when the action executed three hops away from the human? The article argues for receipts that make "I didn't authorize that" a verifiable claim
- bulatb 9mo ago
  There's nothing to prove. Responsibility means you accept the consequences for its actions, whatever they are. You own the benefit? You own the risk. If you don't want to be responsible for what a tool that might do anything at all might do, don't use the tool. The other option is admitting that you don't accept responsibility, not looking for a way to be "responsible" but not accountable.
  - tossandthrow 9mo ago
    Sounds good in theory, doesn't work in reality. Had it worked then we would have seen many more CEOs in prison.
    - NoMoreNicksLeft 9mo ago
      The veil of liability is built into statute, and it's no accident. Such so magic forcefield exists for you, though.
    - walt_grata 9mo ago
      There being a few edge cases where it doesn't work in doesn't mean it doesn't work in the majority of cases and that we shouldn't try to fix the edge cases.
    - bulatb 9mo ago
      We're taking about different things. To take responsibility is volunteering to accept accountability without a fight. In practice, almost everyone is held potentially or actually accountable for things they never had a choice in. Some are never held accountable for things they freely choose, because they have some way to dodge accountability. The CEOs who don't accept accountability were lying when they said they were responsible.
doctorpangloss 9mo ago
Wait till you find out about “pedal confusion.”
nerdsniper 9mo ago
The distinction some people are making is between copy/pasting text vs agentic action. Generally mistakes "work product" as in output from ChatGPT that the human then files with a court, etc. are not forgiven, because if you signed the document, you own its content. Versus some vendor-provided AI Agent which simply takes action on its own that a "reasonable person" would not have expected it to. Often we forgive those kinds of software bloopers.
- ori_b 9mo ago
  If you put a brick on the accelerator of a car and hop out, you don't get to say "I wasn't even in the car when it hit the pedestrian".
  - Shalomboy 9mo ago
    This is true for bricks, but it is not true if your dog starts up your car and hits a pedestrian. Collisions caused by non-human drivers are a fascinating edge case for the times we're in.
    - victorbjorklund 9mo ago
      I don’t know where you from but at least in Sweden you have strict liability for anything your dog does
    - deleted 9mo ago
      [deleted]
    - freejazz 9mo ago
      Prima facie negligence = liability
    - ori_b 9mo ago
      In the USA, at least, it seems pet owners are liable for any harm their pets do.
    - cess11 9mo ago
      Legally, in a lot of jurisdictions, a dog is just your property. What it does, you did, usually with presumed intent or strict liability.
ibejoeb 9mo ago
Yeah. Legal will need to catch up to deal with some things, surely, but the basic principles for this particular scenario aren't that novel. If you're a professional and have an employee acting under your license, there's already liability. There is no warrant concept (not that I can think of right now, at least) that will obviate the need to check the work and carry professional liability insurance. There will always be negligence and bad actors. The new and interesting part is that while we have incentives and deterrents to keep our human agents doing the right thing, there isn't really an analog to check the non-human agent. We don't have robot prison yet.
imiric 9mo ago
That's quickly becoming difficult to determine. The workflow of starting dozens or hundreds of "agents" that work autonomously is starting to gain traction. The goal of people who work like this is to completely automate software development. At some point they want to be able to give the tool an arbitrary task, presumably one that benefits them in some way, and have it build, deploy, and use software to complete it. When millions of people are doing this, and the layers of indirection grow in complexity, how do you trace the result back to a human? Can we say that a human was really responsible for it? Maybe this seems simple today, but the challenges this technology forces on society are numerous, and we're far from ready for it.
- niyikiza 9mo ago
  This is the problem we're working on. When orchestrators spawn sub-agents spawn tools, there's no artifact showing how authority flowed through the chain. Warrants are a primitive for this: signed authorization that attenuates at each hop. Each delegation is signed, scope can only narrow, and the full chain is verifiable at the end. Doesn't matter how many layers deep.
- Wobbles42 9mo ago
  Translation: People want to use a tool and not be liable for the result. People not wanting to be liable for their actions is not new. AI hasn't changed anything here, it's just a new lame excuse.
NedF 9mo ago
[dead]
godelski 9mo ago
> It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not. I hate to ask, but did you RTFA? Scrolling down ever so slightly (emphasis not my own) | *Who authorized this class of action, for which agent identity, under what constraints, for how long; and how did that authority flow?* | A common failure mode in agent incidents is not “we don’t know what happened,” but: | > We can’t produce a crisp artifact showing that a specific human explicitly authorized the scope that made this action possible. They explicitly state that the problem is you don't know which human to point at.
- plastic041 9mo ago
  > They explicitly state that the problem is you don't know which human to point at. The point is "explicitly authorized", as the article emphasizes. It's easy to find who ran the agent(article assumes they have OAuth log). This article is about 'Everyone knows who did it, but did they do it on purpose? Our system can figure it out'
ragBagger 9mo ago
I feel like you have missed the point of this. It isn't to completely absolve the user of liability, it's to prove malice instead of incompetence. If the user claims that they only authorized the bot to review files, but they've warranted the bot to both scan every file and also send emails to outside sources, the competitors in this case, then you now have proof that the user was planning on committing corporate espionage. To use a more sane version of an example below, if your dog runs outside the house and mauls a child, you are obviously guilty of negligence, but if there's proof of you unleashing the dog and ordering the attack, you're guilty of murder.

RobotToaster 9mo ago

If an employee does something during his employment, even if he wasn't told directly to do it, the company can be held vicariously liable, how is this any different?

apercu 9mo ago
I agree with you but you can’t jail a gen-ai model, I guess, is where the difference lies?
- phailhaus 9mo ago
  Nobody tries to jail Microsoft Word, they jail the person using it.
  - gorjusborg 9mo ago
    Nobody tries to jail the automobile being driven when it hits a pedestrian when on cruise control. The driver is responsible for knowing the limits of the tool and adjusting accordingly.
- LeifCarrotson 9mo ago
  "The company can be held vicariously liable" means that in this analogy, the company represents the human who used AI inappropriately, and the employee represents the AI model that did something it wasn't directly told to do.

tboyd47 9mo ago

Anytime someone gives you unverified information, they're asking you to become their guinea pig.

noitpmeder 9mo ago

This is some absolute BS. In the current day and age you are 1000% responsible for the externalities of your use of AI. Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider. Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users. Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!! And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business

ceejayoz 9mo ago
T&Cs aren't ironclad. One in which you sell yourself into slavery, for example, would be illegal in the US. All those "we take no responsibility for the [valet parking|rocks falling off our truck|exploding bottles]" disclaimers are largely attempts to dissuade people from trying. As an example, NY bans liability waivers at paid pools, gyms, etc. The gym will still have you sign one! But they have no enforcement teeth beyond people assuming they're valid. https://codes.findlaw.com/ny/general-obligations-law/gob-sect-5-326/ https://codes.findlaw.com/ny/general-obligations-law/gob-sec...
- noitpmeder 9mo ago
  So I can pass on contact breaches due to bugs in software I maintain due to hallucinations by the AI that I used to write the software?? Absolutely no way. "But the AI wrote the bug." Who cares? It could be you, your relative, your boss, your underling, your counterpart in India, ... Your company provided some reasonable guarantee of service (whether explitly enumerated in a contact or not) and you cannot just blindly pass the buck. Sure, after you've settled your claim with the user, maybe TRY to go after the upstream provider, but good luck. (Extreme example) -- If your company produces a pacemaker dependent on AWS/GCP/... and everyone dies as soon as cloudflare has a routing outage that cascades to your provider, oh boy YOU are fucked, not cloudflare or your hosting provider.

freejazz 9mo ago

What a stupid article from someone that has no idea when liability attaches. It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.

gamblor956 9mo ago

It's not a legal defense at all. Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses. However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output. And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).

rpodraza 9mo ago

What problem is this guy trying to solve? Sorry, but in the end, someone's gonna have to be responsible and it's not gonna be a computer program. Someone approved the program's use, it's no different to any other software. If you know agent can make mistakes then you need to verify everything manually, simple as.

pixl97 9mo ago
While we're a long way off from the day science fiction becomes fact, the world is going to shit itself if a self actionable AI bootstraps and causes havoc.

0xTJ 9mo ago

Why would that be any better of a defense than "that preschooler said that I should do it"? People are responsible for their work.

andrewflnr 9mo ago

How does the old proverb go? > A computer must never make a management decision, because a computer cannot be held accountable.

bitwize 9mo ago

aka the Shaggy Defense for the 2020s.

kazinator 9mo ago

This article is well-written insanity. With no amount of detailed logging makes "the AI did it" a valid excuse. It's just a tool. It's like blaming a loose bolt in a Boeing 737 on "screwdriver did it".

kulahan 9mo ago
If you tighten a bolt using a breaker bar and it fails unexpectedly, it really is the screwdriver’s fault though. Nobody can know if the bolt truly has 70 pounds of torque applied. Maybe it broke at 62, and will fail in 3 flights.

stronglikedan 9mo ago

If one of my reports came to me with that defense, I'd write them up twice. Once for whatever they did wrong, and once for insulting my intelligence and wasting my time with that "defense". On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.

thedudeabides5 9mo ago

a machine can never be held accountable but the person who turned it on can simple as

onoesworkacct 9mo ago

IMO everyone is missing the point of this thing. It's not an auth system or security boundary, it doesn't provide any security guarantees whatsoever, it doesn't do anything. The entire point is to cover a company's derriere should their agentic security apparatus (or lack thereof) fail to prevent malicious prompt injection etc. This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken

deleted 9mo ago

[deleted]

amelius 9mo ago

We should demand money back if an LLM hallucinates. And they should be liable.

jbsimpson 9mo ago
It's a fundamental limitations of LLMS - don't use them if you're worried about hallucinations.

Wobbles42 9mo ago

I don't understand what has changed here. AI is just branding. At the end of the day it's still just people using computer software to do stuff. There is going to be a person who did a thing at the end of the day -- either whoever wrote the software or whoever used the tool. The fact that software inexplicably got unreliable when we started stamping "AI" on the box shouldn't really change anything.

booleandilemma 9mo ago

It's the same thing as saying "It's not my fault, it's a bug in the spreadsheet". The human is ultimately responsible. Heck, it's the same thing as someone saying "the computer made a mistake". Computers don't make mistakes. People do. The hallucination defense is not a defense at all and this is a non-issue. There's nothing to talk about here. And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.

Wobbles42 9mo ago
If the LLMs don't start producing positive revenue soon, the consequences for society will be rather short term.

jbsimpson 9mo ago

Anyone using AI tools should assume that at least 20% of what it produces will be wrong: Missing details, inventing things, basic mistakes. Use AI if being 80% right quickly is fine. Otherwise if you have to do the analysis anyway because accuracy is critical, there's little point to the AI - its too unreliable.

overgard 9mo ago

I'm not sure I understand the point of this. If mistakes (or lies) aren't tolerable in the output, then whoever ran the agent should be responsible for reviewing the output and the resulting actions. I don't see how LLMs can be "responsible" for anything because they don't think in the way we do nor do they have motives in the way we do. The "Hallucination Defense" isn't a defense because end of the day, if you ran it, you're responsible, IMO.

Terr_ 9mo ago

A useful phrase for this when it's an intentional/permitted design: "Accountability Sinks." Or perhaps scapegoating at scale. https://news.ycombinator.com/item?id=43877301 https://news.ycombinator.com/item?id=43877301 - 398 comments https://news.ycombinator.com/item?id=41891694 https://news.ycombinator.com/item?id=41891694 - 308 comments

phs318u 9mo ago

Two things: 1. I vaguely recall that in the early days of Windows, the TOS explicitly told users they were not to use it for certain use-cases and the wording was something like "we don't warranty windows to be good for anything, but we EXPLICITLY do not want you to use Windows to do nuclear foo". I expect that if the big LLM vendors aren't already doing this, they soon will. Kind of like how Fox News claims that some of heads on the screen are not journalists at all but merely entertainers (and you wouldn't take anything they say seriously ergo we're not responsible for stuff that happens as a result of people listening to our heads on the screen). 2. IANAL but I believe that in most legal systems, responsibility requires agency. Leaving aside that we call these things "agents" (which is another thing I suspect will change in the marketing), they do not have agency. As a result they must be considered tools. Tools can be used according to the instructions/TOS, or not. If not - whatever it does is down to you. If used within guidelines - it's the manufacturer. So my conclusion is that the vendors - who have made EPIC bets on getting this tech into the hands of as many folks as possible and making it useful for pretty much anything you can think of - will be faced with a dilemma. At the moment, it seems like they believe that (just like the infamous Ford bean counters) the benefits of not restricting the TOS will far outweigh any consequences from bad things happening. Remains to be seen.

plastic041 9mo ago

This is an advertisement for a 'tenuo warrant'. So, I read its document[0]. Put simply, it works like this: 1. A person orders an AI agent to do A. 2. The agent issues a tenuo warrant for doing A. 3. The agent can now only use the tool to perform A. The article is about that 'warrant' can now be used in case of an incident because it contains information such as 'who ordered the task' and 'what authority was given'. I get the idea. This isn't about whether a person is responsible or not(because of course they are). It's more about whether it was intentional. However... wouldn't it be much easier to just save the prompt log? This article is based entirely on "But the prompt history? Deleted."(from the article) situation. [0]: https://tenuo.dev/concepts https://tenuo.dev/concepts

niyikiza 9mo ago
You've got the model right. And saving prompt logs does help with reconstruction. But warrants aren't just "more audit data." They're an authorization primitive enforced in the critical path: scope and constraints are checked mechanically before the action executes. The receipt is a byproduct. Prompt logs tell you what the model claimed it was doing. A warrant is what the human actually authorized, bound to an agent key, verifiable without trusting the agent runtime. This matters more in multi-agent systems. When Agent A delegates to Agent B, which calls a tool, you want to be able to link that action back to the human who started it. Warrants chain cryptographically. Each hop signs and attenuates. The authorization provenance is in the artifact itself.
- plastic041 9mo ago
  But the AI agent still needs to determine which tool is necessary to mint the warrant. What happens if the agent makes a mistake when making warrant?
  - niyikiza 9mo ago
    A worker agent doesn't mint warrants. It receives them. Either it requests a capability and an issuer approves, or the issuer pushes a scoped warrant when assigning a task. Either way, the issuer signs and the agent can only act within those bounds. At execution time, the "verifier" checks the warrant: valid signatures, attenuation (scope only narrows through delegation), TTL (authority is task-scoped), and that the action fits the constraints. Only then does the call proceed. This is sometimes called the P/Q model: the non-deterministic layer proposes, the deterministic layer decides. The agent can ask for anything. It only gets what's explicitly granted. If the agent asks for the wrong thing, it fails closed. If an overly broad scope is approved, the receipt makes that approval explicit and reviewable.

dietr1ch 9mo ago

Blaming AI and expecting it to cover your ass is like saying that you wrote the code with your elbows and expecting that to make things OK.

michaelmrose 9mo ago

Shouldn't all agentic actions with meaningful outputs of importance like moving $48,000 simply be required to terminate in a human designed or verified output with a human in the loop attestation. Eg a list of transactions that isn't AI generated where the only actions that actually move money must operate on the data displayed in the human designed page. A human looks at this and says yes that is acceptable and becomes reasonable for that action.

krapp 9mo ago
The entire goal of AI is to not have humans in the loop at all. So while that should happen, it won't. They'll just add an extra layer of AI to do the verification.
- michaelmrose 9mo ago
  If AI were good enough to detect hallucinations wouldn't that be built into the AI already?
  - niyikiza 9mo ago
    Exactly ... and that's why I'm skeptical of "AI verifies AI" as the primary safety mechanism. The verifier for moving money should be deterministic: constraints, allowlists, spend limits, invoice/PO matching, etc. The LLM can propose actions, but the execution should be gated by a human/polic-issued scope that's mechanically enforced. That's the whole point: constrain the non-deterministic layer with a deterministic one. [0] [0] https://tenuo.dev/constraints https://tenuo.dev/constraints