3 ms·
Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use th
by amluto 2mo ago
Suppose an automaker creates a BankRobberGym and carefully trains the car to autonomously rob simulated banks because they think someone will pay them to use the car to legally test bank security, but they end up, predictably, training the car to autonomously rob a bank when the driver says “I need some cash - take me to the bank”.
Now a driver gives that instruction and a bank gets robbed. I think it would be odd, to say the least, to say that the automaker just made a tool with legit uses.
In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. I understand that lots of companies think it’s cool to hire red teamers to actually pwn the company hiring them instead of just producing a non-pwning audit, but that doesn’t mean that OpenAI and Anthropic should be playing that particular game.
Years ago, I used to have fun finding vulnerabilities in the Linux kernel, and I found quite a few, including a real juicy one that affected FreeBSD as well. But I mostly didn’t even try to write actual weaponized exploits. Partially because I’m just not that interested in the exercise of weaponizing them and partially because I didn’t and still don’t feel that weaponizing them serves a legitimate purpose.
(I found a very recent vuln that I bet a “cyber” model could weaponize, and my thought is mostly “WTF.” There is absolutely no value to society in weaponizing it. The value is in fixing it, which I did.)
Compare this whole mess to companies training self-driving car models. The research groups publishing papers and, presumably, Waymo, create nifty simulated worlds kind of like the “gyms” that LLM trainers use. And you know what the major objective is? Not crashing!
- gruez 2mo agoSo what does this mean for all the hacking competitions (ie. CTFs) for humans? If it turns out one of the attendees went to hack for North Korea should the organizers of the CTF be prosecuted?
- strogonoff 2mo agoThere’s a difference between enabling another individual with free will and agency, and enabling an automated tool (as a bonus, then giving it to the masses & profiting from its use).
- fc417fc802 2mo agoThere are certainly some differences but is one really more ethical than the other? If anything I feel that (for example) manufacturing a gun is much less likely to carry any ethical implications than training someone to use it might.
- strogonoff 2mo agoWell, manufacturing guns is probably heavily regulated…
- gruez 2mo agoI did a cursory search and it seems to be as regulated as restaurants are. There's an application process and on site inspections, but that's about it. The only thing notable is background checks.
- fc417fc802 2mo agoAnd at least in the US if you're a hobbyist at home then there's ~no regulation whatsoever beyond the requirement to permanently affix a serial number and to keep accurate records.
- strogonoff 2mo agoI don’t think anyone is talking about homegrown LLMs, this is commercial products for sale.
- fc417fc802 2mo agoI don't see the difference when it comes to the ethics of making a tool available to the world or teaching someone something. Like who cares if it was a hobbyist versus a professional tutoring outfit that taught someone expressing an open interest in committing terrorism about the chemistry of explosives? The two hypothetical teachers are equally at fault as far as I'm concerned.
- kmoser 2mo ago> In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. A valid reason would be to find those exploit chains so you can fix them. Of course, the model should be sandboxed so that it can't mistakenly exploit live systems.
- otabdeveloper4 2mo ago> In regard to “cyber”, there is, IMO, no valid reason whatsoever to train a model to autonomously create exploit chains. Field testing is a real thing in literally all industries. Except, apparently, the software industry. When it comes to software security and protecting your sensitive data, the solution is "trust me bro, I got my team of the best lawyers on it".
- amluto 2mo ago> Field testing is a real thing in literally all industries. I’m fairly confident that, if a company that makes door locks want to field test their locks, they test the lock and maybe the door. For some reason the software industry likes to hire someone to test the lock but also to bug the conference room, poison the food in the fridge, blackmail the receptionist, and try to intimidate third party vendors into giving away keys to all the other locks, and maybe steal a few cars while they’re at it. I’m not objecting so much to the attempts to exploit one target. I am objecting to the fact that people treat the exploit chains as such a big deal. And the recent models are clearly going massively overboard.