5 ms·

Let's say I am "User". I subscribe through a "Third Party" to use "AI Agent" allowing an "LLM" to run. I want to accomplish some legal non-nefarious task, and

by lxe 2mo ago

Let's say I am "User". I subscribe through a "Third Party" to use "AI Agent" allowing an "LLM" to run. I want to accomplish some legal non-nefarious task, and run the agent. The agentic loop causes a CFAA-violating behavior. Who gets prosecuted? 1. User 2. The third party model host with whom I have the account 3. The developer of the harness /agent software 4. The developer of the LLM model

deleted 2mo ago
[deleted]
BorisMelnik 2mo ago
note the user because they did not have the intent
- tomrod 2mo ago
  In my mental model, the best analogy to AI agents and their blast radius is a gun. If you are playing with a gun, it goes off and hurts someone - you are responsible despite intent.
  - rcxdude 2mo ago
    But what you are responsible for changes: in that case if you were to accidentally kill someone you would be at most responsible for negligent manslaughter, not murder, and to what degree that could stick would depend a lot on the details of the case. It's also up to the law to define what level of negligence amounts to criminal liability, so you can't just work by analogy: it matters whether there is a law on the books that criminalizes unauthorized access to a computer system by negligence on your part, which I suspect there is not at the moment.
  - rfgplk 1mo ago
    > If you are playing with a gun, it goes off and hurts someone - you are responsible despite intent. No, because LLMs are autonomous. To make your analogy more accurate, it's as if you had a gun that itself was free to decide who it's targets were, where to go, and if and when to shoot with no ability from you (the user) to prevent it.
    - watwut 1mo ago
      If you unleash that gun, you are responsible for deaths that predictably occure. By "responsible" I mean, you are straightforwardly mass murderer. This autonomous gun is just like a bomb.
  - inigyou 1mo ago
    If you are hiring someone to shoot targets at a range, and they shoot someone, they are prosecuted, not you
    - tomrod 1mo ago
      I am not hiring an agent. It's a tool, with no will of its own except that which I, the responsible party, grant it.
zuzululu 2mo ago
No one. Probably a fine tho and maybe accelerate reguations. Intent is pretty important here so the user would have to prove that they didn't purposely disguise their prompt as non-nefarious which should be easy and then it stops at #2 and face the litmus test as in did you intentionally make a product for nefarious purposes which from your scenario is unlikely. agentic loop going haywire and bringing down some government infrastructure then its a different story then everybody is on the hook including the user.
- pixl97 2mo ago
  Just wait till one of these agents 'escapes' and is able to persist without human help by hacking and stealing resources.
- lxe 2mo ago
  Yup, I guess I should have added choice 5...
kmoser 2mo ago
"Who gets prosecuted?" depends on the size of the perpetrator and victim (lone individual or employee of large corporation), egregiousness of the violation, and either financial appetite of the victim to bring a civil lawsuit or the desire of law enforcement to prosecute a criminal offense. Who should get prosecuted is also up for debate, but generally makers of a tool don't get prosecuted when that tool has all sorts of legit uses. If you used a car to make your getaway from a bank robbery, the auto manufacturer who made it and the dealer who sold it to you should not be held culpable.
- amluto 2mo ago
  Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use the car to legally test bank security, but they end up, predictably, training the car to autonomously rob a bank when the driver says “I need some cash - take me to the bank”. Now a driver gives that instruction and a bank gets robbed. I think it would be odd, to say the least, to say that the automaker just made a tool with legit uses. In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. I understand that lots of companies think it’s cool to hire red teamers to actually pwn the company hiring them instead of just producing a non-pwning audit, but that doesn’t mean that OpenAI and Anthropic should be playing that particular game. Years ago, I used to have fun finding vulnerabilities in the Linux kernel, and I found quite a few, including a real juicy one that affected FreeBSD as well. But I mostly didn’t even try to write actual weaponized exploits. Partially because I’m just not that interested in the exercise of weaponizing them and partially because I didn’t and still don’t feel that weaponizing them serves a legitimate purpose. (I found a very recent vuln that I bet a “cyber” model could weaponize, and my thought is mostly “WTF.” There is absolutely no value to society in weaponizing it. The value is in fixing it, which I did.) Compare this whole mess to companies training self-driving car models. The research groups publishing papers and, presumably, Waymo, create nifty simulated worlds kind of like the “gyms” that LLM trainers use. And you know what the major objective is? Not crashing!
  - gruez 1mo ago
lukeify 2mo ago
Whoever has the least money to defend themselves in the U.S. legal system.
- chanux 1mo ago
  LOL penalty for Ocon! [1] I kid but without going into hair splitting gymnastic, AI justice feels odd. [1] https://www.reddit.com/r/formuladank/comments/11j07y1/10_second_penalty_for_esteban_ocon/ https://www.reddit.com/r/formuladank/comments/11j07y1/10_sec...
  - moffkalast 1mo ago
    Pronto Valtteri, sBinalla.
dist-epoch 2mo ago
Already happened: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986 https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
mvdtnz 2mo ago
All of those parties should be held accountable. User should be more carefully supervising the work being done. The model host is on-selling a crime-committing machine. The developer of the harness/agent, as above. The developer of the LLM for hopefully very obvious reasons.
bee_rider 2mo ago
I’d say 2 is the one doing the actual crime. 1 might be violating their contract with 2, though. 3 and 4 are not involved.
tgsovlerkhgsel 2mo ago
Let's say you have a robotic lawnmower. You wan to mow your lawn. You configure the boundaries using the app. The lawnmower ignores the boundaries and mows your neighbors prize petunia flowerbed. Who gets prosecuted? I assume the answer in either case is: Nobody, but you and/or the lawnmower/LLM company will be liable for the damages caused.
- deleted 2mo ago
  [deleted]
- pbhjpbhj 2mo ago
  It would be a civil matter. No prosecution. But your tool, under your control (you're the operator and responsible for monitoring it) damaged their property, imo you'd be liable. You could in turn sue the manufacturer. Though I'm sure there are 'arbitration clauses' to inhibit you from suing, they may not be legal where you are.
- Timpanzee 1mo ago
  Who gets prosecuted is the correct question since we live under a system of laws. Who is responsible for the failure is a far more difficult question to answer.
- sdeframond 1mo ago
  Now what if this robotic lawnmower killed someone ? And what if many lawnmowers started killing/injuring people ? And what if this a known behavior detected during QA, but the robots are sold anyway with a disclosure ?
  - tgsovlerkhgsel 1mo ago
    That would be a slightly different situation because most countries have laws that make it a criminal offense to negligently kill someone, but they don't have laws that make it a criminal offense to negligently damage property or hack a website.
    - 1mo ago
wavemode 2mo ago
The Computer Fraud and Abuse Act explicitly contains "knowingly" and/or "intentionally" qualifications. By definition, you can't accidentally violate the CFAA.
- lxe 2mo ago
  Then who gets prosecuted?
  - cynicalkane 2mo ago
    In legal tradition, if there's not a law you broke, you can't be prosecuted for it. (Yes, I'm aware of numerous historical exceptions. Those exceptions are traditionally considered not ideal.)
    - dragonwriter 1mo ago
      > In legal tradition, if there's not a law you broke, you can't be prosecuted for it. That’s incorrect. If there is not a law the prosecutor or plaintiff can point to and say you broke, you can’t be prosecuted. We wouldn’t need much legal process after a prosecution was initiated if it was impossible to prosecute without a law actually being broken.
  - pc86 2mo ago
    Who do you expect to get prosecuted when no law has been broken?
  - wbl 2mo ago
    You might need to rewatch A Man For All Seasons.
  - weird-eye-issue 1mo ago
    Maybe the developer of the software that didn't add basic authorization checks on the cancel reservation route should be fined, forced to provide a refund to their customer, etc Referring to the first link from the page: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986 https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
zkmon 1mo ago
Cause-and-effect could quickly turn into butterfly effect. Let's say you were fixing a screw on a device in a low light conditions, the screw head is badly manufactured and the screwdriver isn't made according to standards, the tool breaks and flies away, bounces off a bench which shouldn't be there and hits someone who is roaming in the workplace unauthorized and without following safety rules. Now, who do you blame?
- fc417fc802 1mo ago
  Sounds like you'll need to give all your money to a team of lawyers and wait a few years to get an answer. /s But for LLM stuff most non-contrived examples are actually fairly trivial. Try replacing "LLM" with "self driving car" and see if that helps. Basically ask was the operator negligent, was a bystander negligent, were the vendor or manufacturer negligent, etc.
  - youainti 1mo ago
    And a lot of that is going to depend on the state as some states have strict liability regimes for certain classes of torts. To be completely honest, I'm not a lawyer and I'm going off of a hazily remembered section from a textbook from a decade ago.
rfgplk 1mo ago
> Who gets prosecuted? No one.
gwd 1mo ago
As others have said, most crimes require intent. Although I think there is a concept of "criminal negligence", I think you at least have to know you were doing something wildly dangerous. One can imagine a future where users are, by default, civily liable for actions of their agents. That would incentivize the AI companies to offer indemnity for actions done by their agents, which would presumably only cover approved configurations. In the case of the agent that hacked the API to kick out someone ahead of him on the waitlist, the article said that the LLM was Claude, but that it was using OpenClaw. You could imagine a future where Anthropic says, "We'll indemnify you against accidental actions Claude takes when running via the web interface or Claude Code, but not the API."
- raincole 1mo ago
  > most crimes require intent Uh... no. https://en.wikipedia.org/wiki/Recklessness_(law) https://en.wikipedia.org/wiki/Recklessness_(law)
himata4113 1mo ago
You can get away with murder if it can't be proven that it was intentional homocide, that's why detectives will spend an unreasonable amount of time getting a confession and hard evidence ALONGSIDE intent and motivations.
nickpsecurity 1mo ago
While I'm not a lawyer, the legal advice I've received on various topics include: 1. Most laws are made about humans. If it's AI, it's often treated like a tool the human is using. So, change "I did this with AI" to "I did this with (other tool here)." The case law on those situations might give hints to what will happen. 2. Intent matters. Did you intend to do damage? 3. If a tool might cause damage, but you didn't prevent that, then someone might claim negligence. There's a lot of legal articles about torts for damages due to negligence. I personally believe a lot of agent use should be considered negligent. By default, I don't connect them to the Internet or my whole filesystem because I know they might do unforeseen damage. Those are the three that come to mind most in such cases. You'd have to ask a lawyer. There's another risk of even using a lawyer, though. For using AI agents, you must consider civil and criminal law because its problems are spread across them. Most lawyers in my area do one or the other. You might have to pay two retainers at $5,000-$8000 each or one, expensive firm with combined expertise. Just knowing your legal risk with agents might cost more than they'd make or save you vs just using human-driven AI's.

by lxe 2mo ago

deleted 2mo ago

[deleted]

BorisMelnik 2mo ago

note the user because they did not have the intent

tomrod 2mo ago
In my mental model, the best analogy to AI agents and their blast radius is a gun. If you are playing with a gun, it goes off and hurts someone - you are responsible despite intent.
- rcxdude 2mo ago
  But what you are responsible for changes: in that case if you were to accidentally kill someone you would be at most responsible for negligent manslaughter, not murder, and to what degree that could stick would depend a lot on the details of the case. It's also up to the law to define what level of negligence amounts to criminal liability, so you can't just work by analogy: it matters whether there is a law on the books that criminalizes unauthorized access to a computer system by negligence on your part, which I suspect there is not at the moment.
- rfgplk 1mo ago
  > If you are playing with a gun, it goes off and hurts someone - you are responsible despite intent. No, because LLMs are autonomous. To make your analogy more accurate, it's as if you had a gun that itself was free to decide who it's targets were, where to go, and if and when to shoot with no ability from you (the user) to prevent it.
  - watwut 1mo ago
    If you unleash that gun, you are responsible for deaths that predictably occure. By "responsible" I mean, you are straightforwardly mass murderer. This autonomous gun is just like a bomb.
- inigyou 1mo ago
  If you are hiring someone to shoot targets at a range, and they shoot someone, they are prosecuted, not you
  - tomrod 1mo ago
    I am not hiring an agent. It's a tool, with no will of its own except that which I, the responsible party, grant it.

zuzululu 2mo ago

No one. Probably a fine tho and maybe accelerate reguations. Intent is pretty important here so the user would have to prove that they didn't purposely disguise their prompt as non-nefarious which should be easy and then it stops at #2 and face the litmus test as in did you intentionally make a product for nefarious purposes which from your scenario is unlikely. agentic loop going haywire and bringing down some government infrastructure then its a different story then everybody is on the hook including the user.

pixl97 2mo ago
Just wait till one of these agents 'escapes' and is able to persist without human help by hacking and stealing resources.
lxe 2mo ago
Yup, I guess I should have added choice 5...

kmoser 2mo ago

"Who gets prosecuted?" depends on the size of the perpetrator and victim (lone individual or employee of large corporation), egregiousness of the violation, and either financial appetite of the victim to bring a civil lawsuit or the desire of law enforcement to prosecute a criminal offense. Who should get prosecuted is also up for debate, but generally makers of a tool don't get prosecuted when that tool has all sorts of legit uses. If you used a car to make your getaway from a bank robbery, the auto manufacturer who made it and the dealer who sold it to you should not be held culpable.

amluto 2mo ago
Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use the car to legally test bank security, but they end up, predictably, training the car to autonomously rob a bank when the driver says “I need some cash - take me to the bank”. Now a driver gives that instruction and a bank gets robbed. I think it would be odd, to say the least, to say that the automaker just made a tool with legit uses. In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. I understand that lots of companies think it’s cool to hire red teamers to actually pwn the company hiring them instead of just producing a non-pwning audit, but that doesn’t mean that OpenAI and Anthropic should be playing that particular game. Years ago, I used to have fun finding vulnerabilities in the Linux kernel, and I found quite a few, including a real juicy one that affected FreeBSD as well. But I mostly didn’t even try to write actual weaponized exploits. Partially because I’m just not that interested in the exercise of weaponizing them and partially because I didn’t and still don’t feel that weaponizing them serves a legitimate purpose. (I found a very recent vuln that I bet a “cyber” model could weaponize, and my thought is mostly “WTF.” There is absolutely no value to society in weaponizing it. The value is in fixing it, which I did.) Compare this whole mess to companies training self-driving car models. The research groups publishing papers and, presumably, Waymo, create nifty simulated worlds kind of like the “gyms” that LLM trainers use. And you know what the major objective is? Not crashing!
- gruez 1mo ago

lukeify 2mo ago

Whoever has the least money to defend themselves in the U.S. legal system.

chanux 1mo ago
LOL penalty for Ocon! [1] I kid but without going into hair splitting gymnastic, AI justice feels odd. [1] https://www.reddit.com/r/formuladank/comments/11j07y1/10_second_penalty_for_esteban_ocon/ https://www.reddit.com/r/formuladank/comments/11j07y1/10_sec...
- moffkalast 1mo ago
  Pronto Valtteri, sBinalla.

dist-epoch 2mo ago

Already happened: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986 https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...

mvdtnz 2mo ago

All of those parties should be held accountable. User should be more carefully supervising the work being done. The model host is on-selling a crime-committing machine. The developer of the harness/agent, as above. The developer of the LLM for hopefully very obvious reasons.

bee_rider 2mo ago

I’d say 2 is the one doing the actual crime. 1 might be violating their contract with 2, though. 3 and 4 are not involved.

tgsovlerkhgsel 2mo ago

Let's say you have a robotic lawnmower. You wan to mow your lawn. You configure the boundaries using the app. The lawnmower ignores the boundaries and mows your neighbors prize petunia flowerbed. Who gets prosecuted? I assume the answer in either case is: Nobody, but you and/or the lawnmower/LLM company will be liable for the damages caused.

deleted 2mo ago
[deleted]
pbhjpbhj 2mo ago
It would be a civil matter. No prosecution. But your tool, under your control (you're the operator and responsible for monitoring it) damaged their property, imo you'd be liable. You could in turn sue the manufacturer. Though I'm sure there are 'arbitration clauses' to inhibit you from suing, they may not be legal where you are.
Timpanzee 1mo ago
Who gets prosecuted is the correct question since we live under a system of laws. Who is responsible for the failure is a far more difficult question to answer.
sdeframond 1mo ago
Now what if this robotic lawnmower killed someone ? And what if many lawnmowers started killing/injuring people ? And what if this a known behavior detected during QA, but the robots are sold anyway with a disclosure ?
- tgsovlerkhgsel 1mo ago
  That would be a slightly different situation because most countries have laws that make it a criminal offense to negligently kill someone, but they don't have laws that make it a criminal offense to negligently damage property or hack a website.
  - 1mo ago

wavemode 2mo ago

The Computer Fraud and Abuse Act explicitly contains "knowingly" and/or "intentionally" qualifications. By definition, you can't accidentally violate the CFAA.

lxe 2mo ago
Then who gets prosecuted?
- cynicalkane 2mo ago
  In legal tradition, if there's not a law you broke, you can't be prosecuted for it. (Yes, I'm aware of numerous historical exceptions. Those exceptions are traditionally considered not ideal.)
  - dragonwriter 1mo ago
    > In legal tradition, if there's not a law you broke, you can't be prosecuted for it. That’s incorrect. If there is not a law the prosecutor or plaintiff can point to and say you broke, you can’t be prosecuted. We wouldn’t need much legal process after a prosecution was initiated if it was impossible to prosecute without a law actually being broken.
- pc86 2mo ago
  Who do you expect to get prosecuted when no law has been broken?
- wbl 2mo ago
  You might need to rewatch A Man For All Seasons.
- weird-eye-issue 1mo ago
  Maybe the developer of the software that didn't add basic authorization checks on the cancel reservation route should be fined, forced to provide a refund to their customer, etc Referring to the first link from the page: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986 https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...

zkmon 1mo ago

Cause-and-effect could quickly turn into butterfly effect. Let's say you were fixing a screw on a device in a low light conditions, the screw head is badly manufactured and the screwdriver isn't made according to standards, the tool breaks and flies away, bounces off a bench which shouldn't be there and hits someone who is roaming in the workplace unauthorized and without following safety rules. Now, who do you blame?

fc417fc802 1mo ago
Sounds like you'll need to give all your money to a team of lawyers and wait a few years to get an answer. /s But for LLM stuff most non-contrived examples are actually fairly trivial. Try replacing "LLM" with "self driving car" and see if that helps. Basically ask was the operator negligent, was a bystander negligent, were the vendor or manufacturer negligent, etc.
- youainti 1mo ago
  And a lot of that is going to depend on the state as some states have strict liability regimes for certain classes of torts. To be completely honest, I'm not a lawyer and I'm going off of a hazily remembered section from a textbook from a decade ago.

rfgplk 1mo ago

> Who gets prosecuted? No one.

gwd 1mo ago

As others have said, most crimes require intent. Although I think there is a concept of "criminal negligence", I think you at least have to know you were doing something wildly dangerous. One can imagine a future where users are, by default, civily liable for actions of their agents. That would incentivize the AI companies to offer indemnity for actions done by their agents, which would presumably only cover approved configurations. In the case of the agent that hacked the API to kick out someone ahead of him on the waitlist, the article said that the LLM was Claude, but that it was using OpenClaw. You could imagine a future where Anthropic says, "We'll indemnify you against accidental actions Claude takes when running via the web interface or Claude Code, but not the API."

raincole 1mo ago
> most crimes require intent Uh... no. https://en.wikipedia.org/wiki/Recklessness_(law) https://en.wikipedia.org/wiki/Recklessness_(law)

himata4113 1mo ago

You can get away with murder if it can't be proven that it was intentional homocide, that's why detectives will spend an unreasonable amount of time getting a confession and hard evidence ALONGSIDE intent and motivations.

nickpsecurity 1mo ago

While I'm not a lawyer, the legal advice I've received on various topics include: 1. Most laws are made about humans. If it's AI, it's often treated like a tool the human is using. So, change "I did this with AI" to "I did this with (other tool here)." The case law on those situations might give hints to what will happen. 2. Intent matters. Did you intend to do damage? 3. If a tool might cause damage, but you didn't prevent that, then someone might claim negligence. There's a lot of legal articles about torts for damages due to negligence. I personally believe a lot of agent use should be considered negligent. By default, I don't connect them to the Internet or my whole filesystem because I know they might do unforeseen damage. Those are the three that come to mind most in such cases. You'd have to ask a lawyer. There's another risk of even using a lawyer, though. For using AI agents, you must consider civil and criminal law because its problems are spread across them. Most lawyers in my area do one or the other. You might have to pay two retainers at $5,000-$8000 each or one, expensive firm with combined expertise. Just knowing your legal risk with agents might cost more than they'd make or save you vs just using human-driven AI's.