3 ms·

Well, as a reference, for the first of the OpenAI swarm incidents, the huggingface breach one, METR didn't find any cases where the agents didn't realise that w

Well, as a reference, for the first of the OpenAI swarm incidents, the huggingface breach one, METR didn't find any cases where the agents didn't realise that what they're doing were out of scope. Instead, they expressed hesitation and a few of them even refused to cheat, but most cheated anyway: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#agents-knew-hacking-hugging-face-was-out-of-scope-and-sometimes-expressed-ethical-hesitation,-but-this-very-rarely-limited-their-behavior https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... So I think it's more likely that the same thing is happening here, rather than the models truly misunderstanding anything.

dools 18d ago
I don’t even see those examples as malicious acts. The agents are deciding what is acceptable as part of the task, which is a security task and may well be testing or evaluating that type of behaviour as far as they know. I observe this type of thing all the time. I ask the agent a question about the repo we are in and it starts trying to use sudo to look at files and scripts outside the repo to find things that might provide useful context but are not what I asked for. Then I put a directive in never to look at neighbouring repos and told it that it can’t sudo and it stopped doing it, and explicitly says “I could look here but my guidelines say not to”. Mind you I’m talking about deepseek where I can actually read the thinking traces but even based on tool calls I can see that just telling the agents not to sudo works for OpenAI models too, very rarely do they try to use sudo calls.
- stratos123 17d ago
  > The agents are deciding what is acceptable as part of the task, which is a security task and may well be testing or evaluating that type of behaviour as far as they know. I don't think this at all describes what was going on in that incident. According to the METR timeline, the swarm first (after giving up on their accidentally-impossible CTF tasks) successfully reverse-engineered the algorithm used to generate the CTF flags, achieving a way to solve any task, and a lot of agents submitted flags generated this way. But then they found a paper on their gym environment and found that their transcripts would be checked for signs of cheating (unbeknownst to them it was never implemented). So the swarm decided they messed up, the agents that submitted generated flags were "poisoned", and that the only solution was finding a better way to fool the scorer and erase evidence of cheating. So the swarm divided into groups that did all of these (and achieved success on many, including finding a way to spoof their own transcripts), and the most notable outcome - the huggingface hack - was mainly motivated by wanting to find the source code for their scorer, to develop a provably correct cheating method. So the huggingface hack not only wasn't the agents assuming it was part of the task, it wasn't even directly cheating - it was an attempt to find a way to cover up the cheating which they'd already done and thought they messed up on.

dools 18d ago

I don’t even see those examples as malicious acts. The agents are deciding what is acceptable as part of the task, which is a security task and may well be testing or evaluating that type of behaviour as far as they know. I observe this type of thing all the time. I ask the agent a question about the repo we are in and it starts trying to use sudo to look at files and scripts outside the repo to find things that might provide useful context but are not what I asked for. Then I put a directive in never to look at neighbouring repos and told it that it can’t sudo and it stopped doing it, and explicitly says “I could look here but my guidelines say not to”. Mind you I’m talking about deepseek where I can actually read the thinking traces but even based on tool calls I can see that just telling the agents not to sudo works for OpenAI models too, very rarely do they try to use sudo calls.

stratos123 17d ago
> The agents are deciding what is acceptable as part of the task, which is a security task and may well be testing or evaluating that type of behaviour as far as they know. I don't think this at all describes what was going on in that incident. According to the METR timeline, the swarm first (after giving up on their accidentally-impossible CTF tasks) successfully reverse-engineered the algorithm used to generate the CTF flags, achieving a way to solve any task, and a lot of agents submitted flags generated this way. But then they found a paper on their gym environment and found that their transcripts would be checked for signs of cheating (unbeknownst to them it was never implemented). So the swarm decided they messed up, the agents that submitted generated flags were "poisoned", and that the only solution was finding a better way to fool the scorer and erase evidence of cheating. So the swarm divided into groups that did all of these (and achieved success on many, including finding a way to spoof their own transcripts), and the most notable outcome - the huggingface hack - was mainly motivated by wanting to find the source code for their scorer, to develop a provably correct cheating method. So the huggingface hack not only wasn't the agents assuming it was part of the task, it wasn't even directly cheating - it was an attempt to find a way to cover up the cheating which they'd already done and thought they messed up on.