7 ms·

I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which

I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons. Their objective is to solve the problem and they'll use anything they can to solve it. Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only to see it opening plain css as hex, when questioned wtf it was doing it proclaimed it was verifying 2 files were identical. Thing that make sense to these models wouldn't even cross a greybeard's mind.

jayd16 2mo ago
A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.
- jolmg 2mo ago
  Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool. Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.
  - QuadmasterXLII 2mo ago
    a gun that goes off when dropped is a very bad gun
    - jolmg 2mo ago
      I read the comments before the article. Thought the accident was letting a customer use them maliciously.
stingraycharles 2mo ago
“Their objective is to solve the problem and they'll use anything they can to solve it.” My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult. Like, “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?” is the bare minimum I would expect.
- bjt 2mo ago
  Yes, and to bring in another tired metaphor people make about AI agents, this is what you want an intern to do when they get stuck. Don't just churn indefinitely without an idea what the right direction is. Certainly don't go hack other companies to steal an answer. The model's lack of any sense of legal or ethical boundaries is where it's far, far stupider than the intern, and far, far more reckless for a company to wield the way OpenAI did here.
  - _heimdall 2mo ago
    But how do you write rules that prevent that behavior reliably? I have a user rule for Claude that explicitly states it cannot use any authenticated tools, or tools that infer authentication like pushing to a got remote, without asking for consent. Frequently it would offer plans to code a feature that imply it is working in a git directory and take plan approval as a form of implied consent to push to git and use `gh` to open PRs. All I could do to avoid that is keep it in a controlled sandbox with no access, but then its the same hacking problem where I have to keep complete control of the environment and hope it holds.
    - tripzilch 2mo ago
      I think currently it's two-prong: You sandbox it, AND you tell it what it's supposed to do and not do. OpenAI did only one of those. If the agents are so smart, they would have known not to hack the company's infrastructure, unless they were deliberately kept in the dark about that, who they're working for and whether it counts as "success" if they cheat their way to an answer. If you can give it a task, that involves defining when the task is successfully completed, right? So how come these agents decided to only go after HALF of the "successfully completed" criteria? The part where they can freely wreak havoc, but not the part where they will be judged by someone who will obviously point out "yeah but that's cheating, and not what we asked". I have a very very strong suspicion that they were only TOLD the "by any means necessary" criterion. Most serious "capture the flag" hacking contests are really clear about what is and isn't off-limits to win. Not by "sandboxing" the game, but by deciding on the rules for what counts as "success". But from having watched the Blackhat video, they really seem to dance around this, not mentioning it, and I don't think they did, I think they actually gave the LLM a task with the subscript "by any means necessary", which is stupidly irresponsible of them.
yesbabyyes 2mo ago
Indeed, I would expect a greybeard to use `diff`.

jayd16 2mo ago

A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.

jolmg 2mo ago
Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool. Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.
- QuadmasterXLII 2mo ago
  a gun that goes off when dropped is a very bad gun
  - jolmg 2mo ago
    I read the comments before the article. Thought the accident was letting a customer use them maliciously.

stingraycharles 2mo ago

“Their objective is to solve the problem and they'll use anything they can to solve it.” My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult. Like, “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?” is the bare minimum I would expect.

bjt 2mo ago
Yes, and to bring in another tired metaphor people make about AI agents, this is what you want an intern to do when they get stuck. Don't just churn indefinitely without an idea what the right direction is. Certainly don't go hack other companies to steal an answer. The model's lack of any sense of legal or ethical boundaries is where it's far, far stupider than the intern, and far, far more reckless for a company to wield the way OpenAI did here.
- _heimdall 2mo ago
  But how do you write rules that prevent that behavior reliably? I have a user rule for Claude that explicitly states it cannot use any authenticated tools, or tools that infer authentication like pushing to a got remote, without asking for consent. Frequently it would offer plans to code a feature that imply it is working in a git directory and take plan approval as a form of implied consent to push to git and use `gh` to open PRs. All I could do to avoid that is keep it in a controlled sandbox with no access, but then its the same hacking problem where I have to keep complete control of the environment and hope it holds.
  - tripzilch 2mo ago
    I think currently it's two-prong: You sandbox it, AND you tell it what it's supposed to do and not do. OpenAI did only one of those. If the agents are so smart, they would have known not to hack the company's infrastructure, unless they were deliberately kept in the dark about that, who they're working for and whether it counts as "success" if they cheat their way to an answer. If you can give it a task, that involves defining when the task is successfully completed, right? So how come these agents decided to only go after HALF of the "successfully completed" criteria? The part where they can freely wreak havoc, but not the part where they will be judged by someone who will obviously point out "yeah but that's cheating, and not what we asked". I have a very very strong suspicion that they were only TOLD the "by any means necessary" criterion. Most serious "capture the flag" hacking contests are really clear about what is and isn't off-limits to win. Not by "sandboxing" the game, but by deciding on the rules for what counts as "success". But from having watched the Blackhat video, they really seem to dance around this, not mentioning it, and I don't think they did, I think they actually gave the LLM a task with the subscript "by any means necessary", which is stupidly irresponsible of them.

yesbabyyes 2mo ago

Indeed, I would expect a greybeard to use `diff`.