4 ms·

It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence. We had OpenAI "accidentally" run a

by zmmmmm 12d ago

It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence. We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't. So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.

monster_truck 12d ago
I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra
maxgashkov 12d ago
Look at the post-incident investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
includenotfound 12d ago
The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.
- zmmmmm 12d ago
  yes, that is the kicker These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.
- zahlman 12d ago
  I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.
  - BLKNSLVR 12d ago
    It's expensive if it wasn't part of the planning and design. The same as 'security' is expensive, or compliance with regulations is expensive. It is also a choice to not do any or all of the above.
  - stephbook 12d ago
    Agents need packages like the rest of us. Ruby gems, npm packages, Maven, pip, docker images.. Not surprised this is always what they have and hack. Who would use an Agent that spends $10,000 re-implementing some OAuth lib or reverse-engineering a proprietary lib when it's free on the internet?
    - tancop 12d ago
      You don't need a full air gap. Set up a microVM with network access limited to local network and send all package requests through a filtering gateway that only allows normal download endpoints. Or self host a big collection of popular packages if you need extra security.
      - amouat 12d ago
        Isn't that exactly what they did? The bots could only access the jfrog instance, so they hacked jfrog?
        exfalso 11d ago
        No that's not what they did, they exposed jfrog raw. It would have been so extremely simple to gate services they need the llm to access... I mean, jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
- verdverm 12d ago
  > You'd think, if they truly believed the model is so dangerous... They would have been watching what it does, especially when running it on ExploitGym of all benchmarks... that is criminal worthy neglegence
BLKNSLVR 12d ago
Sounds like advertising platforms. Spraying malware and links to scam sites all over the place. "They" don't care about the end-people. "They" care about maximising their profit thing, in a vacuum.
slipperybeluga 12d ago
[dead]
swamp-agr 11d ago
What would be the next actionable step for bringing these executives to the Congress for further questioning?
deleted 11d ago
[deleted]

by zmmmmm 12d ago

monster_truck 12d ago

I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra

maxgashkov 12d ago

Look at the post-incident investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.

includenotfound 12d ago

The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.

zmmmmm 12d ago
yes, that is the kicker These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.
zahlman 12d ago
I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think. I still think this is a sign that they are not taking their own rhetoric seriously.
- BLKNSLVR 12d ago
  It's expensive if it wasn't part of the planning and design. The same as 'security' is expensive, or compliance with regulations is expensive. It is also a choice to not do any or all of the above.
- stephbook 12d ago
  Agents need packages like the rest of us. Ruby gems, npm packages, Maven, pip, docker images.. Not surprised this is always what they have and hack. Who would use an Agent that spends $10,000 re-implementing some OAuth lib or reverse-engineering a proprietary lib when it's free on the internet?
  - tancop 12d ago
    You don't need a full air gap. Set up a microVM with network access limited to local network and send all package requests through a filtering gateway that only allows normal download endpoints. Or self host a big collection of popular packages if you need extra security.
    - amouat 12d ago
      Isn't that exactly what they did? The bots could only access the jfrog instance, so they hacked jfrog?
      - exfalso 11d ago
        No that's not what they did, they exposed jfrog raw. It would have been so extremely simple to gate services they need the llm to access... I mean, jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
verdverm 12d ago
> You'd think, if they truly believed the model is so dangerous... They would have been watching what it does, especially when running it on ExploitGym of all benchmarks... that is criminal worthy neglegence

BLKNSLVR 12d ago

Sounds like advertising platforms. Spraying malware and links to scam sites all over the place. "They" don't care about the end-people. "They" care about maximising their profit thing, in a vacuum.

slipperybeluga 12d ago

[dead]

swamp-agr 11d ago

What would be the next actionable step for bringing these executives to the Congress for further questioning?

deleted 11d ago

[deleted]