3 ms·

"Who gets prosecuted?" depends on the size of the perpetrator and victim (lone individual or employee of large corporation), egregiousness of the violation, and

by kmoser 2mo ago

"Who gets prosecuted?" depends on the size of the perpetrator and victim (lone individual or employee of large corporation), egregiousness of the violation, and either financial appetite of the victim to bring a civil lawsuit or the desire of law enforcement to prosecute a criminal offense. Who should get prosecuted is also up for debate, but generally makers of a tool don't get prosecuted when that tool has all sorts of legit uses. If you used a car to make your getaway from a bank robbery, the auto manufacturer who made it and the dealer who sold it to you should not be held culpable.

amluto 2mo ago
Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use the car to legally test bank security, but they end up, predictably, training the car to autonomously rob a bank when the driver says “I need some cash - take me to the bank”. Now a driver gives that instruction and a bank gets robbed. I think it would be odd, to say the least, to say that the automaker just made a tool with legit uses. In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. I understand that lots of companies think it’s cool to hire red teamers to actually pwn the company hiring them instead of just producing a non-pwning audit, but that doesn’t mean that OpenAI and Anthropic should be playing that particular game. Years ago, I used to have fun finding vulnerabilities in the Linux kernel, and I found quite a few, including a real juicy one that affected FreeBSD as well. But I mostly didn’t even try to write actual weaponized exploits. Partially because I’m just not that interested in the exercise of weaponizing them and partially because I didn’t and still don’t feel that weaponizing them serves a legitimate purpose. (I found a very recent vuln that I bet a “cyber” model could weaponize, and my thought is mostly “WTF.” There is absolutely no value to society in weaponizing it. The value is in fixing it, which I did.) Compare this whole mess to companies training self-driving car models. The research groups publishing papers and, presumably, Waymo, create nifty simulated worlds kind of like the “gyms” that LLM trainers use. And you know what the major objective is? Not crashing!
- gruez 2mo ago
  So what does this mean for all the hacking competitions (ie. CTFs) for humans? If it turns out one of the attendees went to hack for North Korea should the organizers of the CTF be prosecuted?
  - strogonoff 2mo ago
    There’s a difference between enabling another individual with free will and agency, and enabling an automated tool (as a bonus, then giving it to the masses & profiting from its use).
    - fc417fc802 1mo ago
      There are certainly some differences but is one really more ethical than the other? If anything I feel that (for example) manufacturing a gun is much less likely to carry any ethical implications than training someone to use it might.
      - strogonoff 1mo ago
        Well, manufacturing guns is probably heavily regulated…
        gruez 1mo ago
        I did a cursory search and it seems to be as regulated as restaurants are. There's an application process and on site inspections, but that's about it. The only thing notable is background checks.
        fc417fc802 1mo ago
        And at least in the US if you're a hobbyist at home then there's ~no regulation whatsoever beyond the requirement to permanently affix a serial number and to keep accurate records.
dfxm12 2mo ago
Your bank robbery situation is not apt to OP's question. It's pretty much the opposite situation. OP suggests a situation where the operator is probably using the tool in good faith but the tool appears to be operating in a faulty manner. For some more context, Toyota faced criminal penalties in the US for their unintended acceleration issues back in 2010.
- kmoser 1mo ago
  OP's scenario specified only "AI Agent" and did not describe how it was trained or what its parameters were supposed to be. You're assuming that the tool was designed such that it couldn't break the law, and therefore if it did, that would be considered faulty behavior, but OP said no such thing. Much rests on whether the user knew, or should have known, whether the tool was capable of actions which could break the law, as well as what steps (if any) the creator of the tool took to ensure the tool was legally compliant, and what warnings they gave to subscribers about possible unintended side-effects. OP specified none of this.
- wbadart 1mo ago
  Slightly off-topic, but I found it interesting to learn that, in spite of the rulung against Toyota, the issue likely wasn't an engineering fault at the end of the day. Here's Malcolm Gladwell's coverage from his podcast a little while ago: https://www.pushkin.fm/podcasts/revisionist-history/blame-game https://www.pushkin.fm/podcasts/revisionist-history/blame-ga...

by kmoser 2mo ago

amluto 2mo ago

Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use the car to legally test bank security, but they end up, predictably, training the car to autonomously rob a bank when the driver says “I need some cash - take me to the bank”. Now a driver gives that instruction and a bank gets robbed. I think it would be odd, to say the least, to say that the automaker just made a tool with legit uses. In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. I understand that lots of companies think it’s cool to hire red teamers to actually pwn the company hiring them instead of just producing a non-pwning audit, but that doesn’t mean that OpenAI and Anthropic should be playing that particular game. Years ago, I used to have fun finding vulnerabilities in the Linux kernel, and I found quite a few, including a real juicy one that affected FreeBSD as well. But I mostly didn’t even try to write actual weaponized exploits. Partially because I’m just not that interested in the exercise of weaponizing them and partially because I didn’t and still don’t feel that weaponizing them serves a legitimate purpose. (I found a very recent vuln that I bet a “cyber” model could weaponize, and my thought is mostly “WTF.” There is absolutely no value to society in weaponizing it. The value is in fixing it, which I did.) Compare this whole mess to companies training self-driving car models. The research groups publishing papers and, presumably, Waymo, create nifty simulated worlds kind of like the “gyms” that LLM trainers use. And you know what the major objective is? Not crashing!

gruez 2mo ago
So what does this mean for all the hacking competitions (ie. CTFs) for humans? If it turns out one of the attendees went to hack for North Korea should the organizers of the CTF be prosecuted?
- strogonoff 2mo ago
  There’s a difference between enabling another individual with free will and agency, and enabling an automated tool (as a bonus, then giving it to the masses & profiting from its use).
  - fc417fc802 1mo ago
    There are certainly some differences but is one really more ethical than the other? If anything I feel that (for example) manufacturing a gun is much less likely to carry any ethical implications than training someone to use it might.
    - strogonoff 1mo ago
      Well, manufacturing guns is probably heavily regulated…
      - gruez 1mo ago
        I did a cursory search and it seems to be as regulated as restaurants are. There's an application process and on site inspections, but that's about it. The only thing notable is background checks.
        fc417fc802 1mo ago
        And at least in the US if you're a hobbyist at home then there's ~no regulation whatsoever beyond the requirement to permanently affix a serial number and to keep accurate records.

dfxm12 2mo ago

Your bank robbery situation is not apt to OP's question. It's pretty much the opposite situation. OP suggests a situation where the operator is probably using the tool in good faith but the tool appears to be operating in a faulty manner. For some more context, Toyota faced criminal penalties in the US for their unintended acceleration issues back in 2010.

kmoser 1mo ago
OP's scenario specified only "AI Agent" and did not describe how it was trained or what its parameters were supposed to be. You're assuming that the tool was designed such that it couldn't break the law, and therefore if it did, that would be considered faulty behavior, but OP said no such thing. Much rests on whether the user knew, or should have known, whether the tool was capable of actions which could break the law, as well as what steps (if any) the creator of the tool took to ensure the tool was legally compliant, and what warnings they gave to subscribers about possible unintended side-effects. OP specified none of this.
wbadart 1mo ago
Slightly off-topic, but I found it interesting to learn that, in spite of the rulung against Toyota, the issue likely wasn't an engineering fault at the end of the day. Here's Malcolm Gladwell's coverage from his podcast a little while ago: https://www.pushkin.fm/podcasts/revisionist-history/blame-game https://www.pushkin.fm/podcasts/revisionist-history/blame-ga...