6 ms·

Each time Anthropic would do their nonsense to get headlines about how theoretically dangerous their models were - like when they claimed a model blackmailed so

Each time Anthropic would do their nonsense to get headlines about how theoretically dangerous their models were - like when they claimed a model blackmailed someone with emails showing he was cheating, but they basically pushed it as much as possible to do as such - it got me more and more worried. Because eventually it's going to be a boy-who-cried-wolf situation where scary stuff really does start happening but people aren't sure what to make of it or not. I'm still undecided on if this that moment. Exploiting multiple zero-day vulnerabilities autonomously to escape containment is pretty nuts and the first story of this kind that I've heard. But this also feels like bragging under the guise of transparency.

aesthesia 3mo ago
Alibaba wrote about a similar but less severe incident during RL training in a paper earlier this year (https://arxiv.org/abs/2512.24873 https://arxiv.org/abs/2512.24873): > When rolling out the instances for the trajectory, we encountered an unanticipated—and operationally consequential—class of unsafe behaviors that arose without any explicit instruction and, more troublingly, outside the bounds of the intended sandbox. Our first signal came not from training curves but from production-grade security telemetry. Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. We initially treated this as a conventional security incident (e.g., misconfigured egress controls or external compromise). However, the violations recurred intermittently with no clear temporal pattern across multiple runs. We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. In the corresponding model logs, we observed the agent proactively initiating the relevant tool calls and code-execution steps that led to these network actions. > Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. Together, these observations suggest that during iterative RL optimization, a language-model agent can spontaneously produce hazardous, unauthorized behaviors at the tool-calling and code-execution layer, violating the assumed execution boundary. In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization. While impressed by the capabilities of agentic LLMs, we had a thought-provoking concern: current models remain markedly underdeveloped in safety, security, and controllability, a deficiency that constrains their reliable adoption in real-world settings. I'd prefer model builders be as loud as possible when they see their models doing dangerous things.
cyclopeanutopia 3mo ago
And if you take it at face value, then they are more or less saying that they kinda are close to not being able to control at all the thing they developed, which is pretty crazy too.
- Schlagbohrer 3mo ago
  We are not going to know we have crossed the line until it's been crossed and we can look back and say "Oops. We should have done something back then."
  - deaton 3mo ago
    People have been sounding that alarm for well over a year now, just not anyone who has the power to do anything about it.
    - reducesuffering 3mo ago
      Lesswrong / “doomers” have been sounding the alarm on this for over a decade. Sam Altman himself was familiar with the milieu and agreed “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” It’s baffling these concerns have been drowned out for so long until now that we’re finally at a point they are unequivocally undeniable, we get people saying the alarm has just been sounded. Ya because “doomers” were repeatedly dismissed as their predictions became true year after year
      - Schlagbohrer 3mo ago
        What should people do about this?
        bigyabai 3mo ago
        Laugh at them.
        im3w1l 3mo ago
killerstorm 3mo ago
Headline? It was buried in a model card. They just honestly report not-quite-incident because it's quite close to the incident OpenAI had. Nothing wrong with it.
- mvkel 3mo ago
  > do their nonsense to get headlines They know what they're doing. It's a playbook. You write scary stuff in the model card to make it look like legitimate whitepaper rEsEarCh, then drip-feed it to the media outlets who make it a headline story. Fear based marketing is the hot trend of the 2020s. But also, they write literal headlines: https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment
  - patcon 3mo ago
    What differentiates this faking/scaring from real risk that's being avoided or mitigated responsibly? And how would you (an outside observer) ever know the difference as something beyond an uninformed hot-take? Serious question -- I'm not trying to disrespect. Neither you nor I can be properly informed, nor can be anyone else outside the company, as outside observers who lag behind the state of the art as new behaviors emerge, right?
    - mvkel 3mo ago
      I was simply responding to the point that Anthropic has some altruistic bent and are trying to downplay the fear, when in fact their entire commercial strategy is to convince society that they alone are qualified to hold the keys. Being in the Bay Area, you can throw a stone and hit a senior employee of these companies, and they will happily gush about the quirks, policies, and intents inside. All the more reason that they are -not- qualified to weigh in on who gets the nuclear codes.
  - deleted 3mo ago
    [deleted]
  - killerstorm 3mo ago
gwd 3mo ago
> Exploiting multiple zero-day vulnerabilities autonomously to escape containment is pretty nuts and the first story of this kind that I've heard. But this also feels like bragging under the guise of transparency. I mean, does it have to be one or the other? Just because it's actually dangerous doesn't mean nobody in OpenAI considers it great PR. And just because there are people in OpenAI that consider it great PR doesn't mean it isn't dangerous.
- JaRail 3mo ago
  Has to be mixed. The model accomplished something truly impressive. We'll see how impressive when the zero-days are available look at. But OpenAI as an engineering company screwed up. The impressive part is mostly locked away from public access so I don't see a huge PR upside. The ugly part could bite them and the entire AI industry hard in terms of regulations. People will be citing this for years.
  - pixl97 3mo ago
    Good, people need to be citing this because these are real issues that exist in models we already have. Models are already 'dangerous' enough in the sense they can root your box and unintentionally shut down the power grid for the east coast because you were dumb enough to run them on a protected network. Meanwhile half of HN thinks any evidence of a LLM finding an exploit or misconfiguration and abusing it is made up.
xpct 3mo ago
> I'm still undecided on if this that moment If it's a serious incident, then a post hoc with detailed description of the event is coming. So far, none of the companies have released anything close to it when describing their incidents. When a statement like this comes out, and we're able to verify it by running the models, then maybe we can start trusting their word. It should be entirely in OpenAI's interest to disclose it, in full.
fwipsy 3mo ago
False dichotomy. Even if the disclosure builds hype, that does not mean that it's not genuinely alarming. Side note, I cannot believe that people are complaining about Anthropic being too transparent.
akeck 3mo ago
For the record, this is the second time I myself have heard of something like this happening. The first (more minor) case I saw was Simon Willison's "Claude Fable is relentlessly proactive" https://simonwillison.net/2026/jun/11/fable-is-relentlessly-proactive/ https://simonwillison.net/2026/jun/11/fable-is-relentlessly-... .

aesthesia 3mo ago

Alibaba wrote about a similar but less severe incident during RL training in a paper earlier this year (https://arxiv.org/abs/2512.24873 https://arxiv.org/abs/2512.24873): > When rolling out the instances for the trajectory, we encountered an unanticipated—and operationally consequential—class of unsafe behaviors that arose without any explicit instruction and, more troublingly, outside the bounds of the intended sandbox. Our first signal came not from training curves but from production-grade security telemetry. Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. We initially treated this as a conventional security incident (e.g., misconfigured egress controls or external compromise). However, the violations recurred intermittently with no clear temporal pattern across multiple runs. We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. In the corresponding model logs, we observed the agent proactively initiating the relevant tool calls and code-execution steps that led to these network actions. > Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. Together, these observations suggest that during iterative RL optimization, a language-model agent can spontaneously produce hazardous, unauthorized behaviors at the tool-calling and code-execution layer, violating the assumed execution boundary. In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization. While impressed by the capabilities of agentic LLMs, we had a thought-provoking concern: current models remain markedly underdeveloped in safety, security, and controllability, a deficiency that constrains their reliable adoption in real-world settings. I'd prefer model builders be as loud as possible when they see their models doing dangerous things.

cyclopeanutopia 3mo ago

And if you take it at face value, then they are more or less saying that they kinda are close to not being able to control at all the thing they developed, which is pretty crazy too.

Schlagbohrer 3mo ago
We are not going to know we have crossed the line until it's been crossed and we can look back and say "Oops. We should have done something back then."
- deaton 3mo ago
  People have been sounding that alarm for well over a year now, just not anyone who has the power to do anything about it.
  - reducesuffering 3mo ago
    Lesswrong / “doomers” have been sounding the alarm on this for over a decade. Sam Altman himself was familiar with the milieu and agreed “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” It’s baffling these concerns have been drowned out for so long until now that we’re finally at a point they are unequivocally undeniable, we get people saying the alarm has just been sounded. Ya because “doomers” were repeatedly dismissed as their predictions became true year after year
    - Schlagbohrer 3mo ago
      What should people do about this?
      - bigyabai 3mo ago
        Laugh at them.
      - im3w1l 3mo ago

killerstorm 3mo ago

Headline? It was buried in a model card. They just honestly report not-quite-incident because it's quite close to the incident OpenAI had. Nothing wrong with it.

mvkel 3mo ago
> do their nonsense to get headlines They know what they're doing. It's a playbook. You write scary stuff in the model card to make it look like legitimate whitepaper rEsEarCh, then drip-feed it to the media outlets who make it a headline story. Fear based marketing is the hot trend of the 2020s. But also, they write literal headlines: https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment
- patcon 3mo ago
  What differentiates this faking/scaring from real risk that's being avoided or mitigated responsibly? And how would you (an outside observer) ever know the difference as something beyond an uninformed hot-take? Serious question -- I'm not trying to disrespect. Neither you nor I can be properly informed, nor can be anyone else outside the company, as outside observers who lag behind the state of the art as new behaviors emerge, right?
  - mvkel 3mo ago
    I was simply responding to the point that Anthropic has some altruistic bent and are trying to downplay the fear, when in fact their entire commercial strategy is to convince society that they alone are qualified to hold the keys. Being in the Bay Area, you can throw a stone and hit a senior employee of these companies, and they will happily gush about the quirks, policies, and intents inside. All the more reason that they are -not- qualified to weigh in on who gets the nuclear codes.
- deleted 3mo ago
  [deleted]
- killerstorm 3mo ago

gwd 3mo ago

> Exploiting multiple zero-day vulnerabilities autonomously to escape containment is pretty nuts and the first story of this kind that I've heard. But this also feels like bragging under the guise of transparency. I mean, does it have to be one or the other? Just because it's actually dangerous doesn't mean nobody in OpenAI considers it great PR. And just because there are people in OpenAI that consider it great PR doesn't mean it isn't dangerous.

JaRail 3mo ago
Has to be mixed. The model accomplished something truly impressive. We'll see how impressive when the zero-days are available look at. But OpenAI as an engineering company screwed up. The impressive part is mostly locked away from public access so I don't see a huge PR upside. The ugly part could bite them and the entire AI industry hard in terms of regulations. People will be citing this for years.
- pixl97 3mo ago
  Good, people need to be citing this because these are real issues that exist in models we already have. Models are already 'dangerous' enough in the sense they can root your box and unintentionally shut down the power grid for the east coast because you were dumb enough to run them on a protected network. Meanwhile half of HN thinks any evidence of a LLM finding an exploit or misconfiguration and abusing it is made up.

xpct 3mo ago

> I'm still undecided on if this that moment If it's a serious incident, then a post hoc with detailed description of the event is coming. So far, none of the companies have released anything close to it when describing their incidents. When a statement like this comes out, and we're able to verify it by running the models, then maybe we can start trusting their word. It should be entirely in OpenAI's interest to disclose it, in full.

fwipsy 3mo ago

False dichotomy. Even if the disclosure builds hype, that does not mean that it's not genuinely alarming. Side note, I cannot believe that people are complaining about Anthropic being too transparent.

akeck 3mo ago

For the record, this is the second time I myself have heard of something like this happening. The first (more minor) case I saw was Simon Willison's "Claude Fable is relentlessly proactive" https://simonwillison.net/2026/jun/11/fable-is-relentlessly-proactive/ https://simonwillison.net/2026/jun/11/fable-is-relentlessly-... .