5 ms·

The Hugging Face Hack Wasn't What It Was Cracked Up to Be

55 points by kgwgk 14d ago

deleted 14d ago
[deleted]
bryan0 14d ago
I would just recommend reading what actually happened: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... I don’t think downplaying what occurred is really beneficial to anyone.
- DrewADesign 14d ago
  If you haven’t read the article, it might present some useful counterpoints. It criticizes that particular report’s portrayal of the incident.
  - bryan0 14d ago
    It really doesn’t though. It criticizes how the news media reported on the incident but this opinion piece is no better. Just read the original source itself.
    - DrewADesign 14d ago
      I did read the report. Yes, this article does criticize the presentation of the events in that report. This article doesn’t say that the report is lying, but it does say that it deliberately sensationalized aspects in unhelpful ways, and de-emphasizes important considerations. That is important when it’s pretty obvious these companies are using fear to signal their products are more powerful than they are.
- m348e912 14d ago
  If you don't like reading, Dwarkesh Patel offers a solid summary of what happened based on the published report. https://www.youtube.com/watch?v=u15N3l4RT80 https://www.youtube.com/watch?v=u15N3l4RT80
  - falaki 14d ago
    How many security incident reports and postmortems has Dwarfish Petal written in his career? Does he understand the basics of system security? Can he tell the difference between secure and insecure configuration? Paint me skeptical.
- ozozozd 14d ago
  Excuse my language but this is the shittiest and the most unserious “report” I’ve ever seen in my life. At best this is a write-up, but it’s more like a juicy pop article. Quotes from agents between paragraphs, referring to the “collective”, “agents participated in the attack” - seriously? It’s clearly written for effect and to stimulate people’s imaginations. If these people are this unserious, P(doom) should go up to 20-30%.
  - bryan0 14d ago
    It is sad that this is the best level of reporting and oversight of these types of events available to us, but it is currently all we have. We have to do better, but to dismiss the report because of the style and tone would be foolish.
    - DrewADesign 14d ago
      And to dismiss criticism of the report’s obvious slant is even more foolish. Just because the report isn’t outright lying about the facts doesn’t mean it’s an objective, definitive, and unimpeachable analysis of what happened. It definitely shouldn’t be the only thing anybody should read on the topic. This article also has a slant. It’s not an objective, definitive, and unimpeachable analysis of what happened, either. It also, definitely, shouldn’t be the only thing anybody should read on the topic.
LoganDark 14d ago
This article is AI generated, but I kinda reject the premise that it's "not all it cracked up to be" just because the agents had reasons to act the way they did & were a result of human error. The hack still happened, it should still be a wake up call, we are going to start seeing this more frequently, and learning to defend against it is going to become more important over time. None of that is challenged by any particular reason for it happening, it still happened and it's still going to happen again. Threat models are going to have to start including that IPv4 (or whatever) scanners aren't necessarily going to only be spray and pray anymore, they could have relentless automated models at the other end that will literally dig into the particulars of your infrastructure looking for novel vulnerabilities to exploit. Maybe people will finally start to understand why security by obscurity has never been very reliable.
- minimaxir 14d ago
  Calling an article from the Wall Street Journal AI generated is a stretch.
  - deleted 14d ago
    [deleted]
  - LoganDark 14d ago
    I don't care who published it, this is clearly AI-written. > But the Hugging Face episode is different. It left logs, reports, design decisions and identifiable points at which human beings could have intervened. And that record suggests a less thrilling but more useful lesson. > People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result "rogue AI" does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
    - nkurz 13d ago
      I think your confidence is likely misplaced, but I vouched for your comment to revive it from the dead because you expressed your opinion clearly and with examples. To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.
woleium 14d ago
Reads like an AI trying to downplay the severity of the situation. Very cyberpunk.
falaki 14d ago
The incident report leaves a lot to be desired. - It was done by two institutes with organizational ties to OpenAI: METR and Redwood Research - METR and Redwood Research are institutional pillars of the "AI Safety" wing of the Effective Altruism movement. This clearly shows their prior biases towards "AI existential risk" rather than technical/engineering root-causing of the incident - If you reed the report, it is not on-par with what you find from other companies. - The access that was given to both was mediated and controlled by OpenAI. It is not clear, if they were able to get to the bottom of engineering flaws. It is not clear if they could see all the audit logs, etc. Considering all the above, I consider the whole episode more of a PR stunt. I understand that is not a majority opinion at this point.
- bryan0 14d ago
  It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt. Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
  - xelxebar 14d ago
    grok make good security? is lot of money and much time. grok not do? easy. maybe good PR too. TIL my brain is an Olympic gymnast. The incentive structures are clearly there. I think the case of intentional manufacture is definitely weaker, requiring the conjunction of more weakly-supported events.
  - majormajor 14d ago
    The bit that makes me cynical about the "dangerous" behavior is that it's not like this was being used by someone else than who made it or operating completely independently outside of its creators. Everything dangerous seems like a direct consequence of risky human choices, starting with knowledge bases used during core training, harnessing and tuning to be task-completion-oriented to a fault + deeply oriented towards using and looking for external tools and resources, and overconfidence in their sandboxing for testing. "We stuffed a bunch of information on how to exploit computer systems into an automaton and told it to go brrrrr until it could answer a question" - this is something intentional done by humans. This is not some "rogue AI" trained to search for cancer cures that instead completely independently decided to hack tech companies. The companies directing things in dangerous directions need to own that they're consciously pushing in those directions.
deleted 14d ago
[deleted]
gz5 14d ago
https://archive.is/hnpmx https://archive.is/hnpmx all 3 can be true at same time: 1. sensationalizing rarely helps and can obscure and hurt 2. the AI capabilities are underrated 3. attempted govt regulation is not the answer the intent, or lack of intent, of the agent is mainly irrelevant if it is in the hands of a human with 'bad' intentions. what is more relevant are the capabilities of human + AI.
- f30e3dfed1c9 14d ago
  "attempted govt regulation is not the answer" Self-regulation? None at all?
  - OutOfHere 14d ago
    There already exist laws against hacking. Legal damages already can apply. Further laws aren't necessary and will only suppress open-weight AI firms, making the monopoly of large AI firms more entrenched. See the first comment (by Brett Matson) on the archive link of the WSJ article.
    - iAMkenough 14d ago
      I think we need an AI sheriff that aggressively enforces the law. The current level of enforcement and the fine schedule is not enough of a deterrent to prevent these large corporations from breaking existing laws. Frequently make an example of them and you’ll be hailed a hero by the general populous.
cm2187 14d ago
The only difference I see on the facts vs what I read so far is that OpenAI knew of the hack and decided not to intervene. I haven't read the report but I find it hard to believe that OpenAI deliberately let its agents hack a third party company during a training run. It would certainly attract a criminal liability, which would be surprising to admit in writing.
robswc 14d ago
I truly just don't believe a word regarding achievements from these frontier labs. It started with that guy from google saying the LLM was "alive" and every time I see a press release from these labs it reads mostly like a marketing scheme. "Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!"
- includenotfound 14d ago
  > "Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!" While simultaneously announcing: "btw we're releasing the ever more powerful and more dangerous and more benchmaxing model next week, get our $200 sub asap!"
lossolo 14d ago
After a recent interview[1] with Noam Brown (OpenAI), in which he said they had specifically trained agents for cooperation before this hack, the hack doesn't seem as impressive anymore. They didn't even bother to control the post training rollouts, so the training data got contaminated and was included in the training of other agents. Connect these two dots and you have the Hugging Face hack. And at the beginning, when these incidents were first reported, it was portrayed as if all of this (the communication between agents etc.) was emergent behaviour. 1. https://www.youtube.com/watch?v=6AgOfiZOWiY https://www.youtube.com/watch?v=6AgOfiZOWiY
- genxy 14d ago
  The result doesn't change, the result is what is dangerous. And the natural ability for the model to form a swarm is now trained in, this is extremely risky. The models should not be able to form swarms, this is an extremely dangerous property. They are basically creating a slime mold or ant colony that can speak multiple languages, create their own language and operate as a collective. The idea that you are impressed is a non sequitur.
spit2wind 14d ago
So, how is any of this legal? If I set up a script that breaks into the systems of a company, isn't that, like, a crime?

55 points by kgwgk 14d ago

deleted 14d ago

[deleted]

I would just recommend reading what actually happened: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... I don’t think downplaying what occurred is really beneficial to anyone.

DrewADesign 14d ago
If you haven’t read the article, it might present some useful counterpoints. It criticizes that particular report’s portrayal of the incident.
- bryan0 14d ago
  It really doesn’t though. It criticizes how the news media reported on the incident but this opinion piece is no better. Just read the original source itself.
  - DrewADesign 14d ago
    I did read the report. Yes, this article does criticize the presentation of the events in that report. This article doesn’t say that the report is lying, but it does say that it deliberately sensationalized aspects in unhelpful ways, and de-emphasizes important considerations. That is important when it’s pretty obvious these companies are using fear to signal their products are more powerful than they are.
m348e912 14d ago
If you don't like reading, Dwarkesh Patel offers a solid summary of what happened based on the published report. https://www.youtube.com/watch?v=u15N3l4RT80 https://www.youtube.com/watch?v=u15N3l4RT80
- falaki 14d ago
  How many security incident reports and postmortems has Dwarfish Petal written in his career? Does he understand the basics of system security? Can he tell the difference between secure and insecure configuration? Paint me skeptical.
ozozozd 14d ago
Excuse my language but this is the shittiest and the most unserious “report” I’ve ever seen in my life. At best this is a write-up, but it’s more like a juicy pop article. Quotes from agents between paragraphs, referring to the “collective”, “agents participated in the attack” - seriously? It’s clearly written for effect and to stimulate people’s imaginations. If these people are this unserious, P(doom) should go up to 20-30%.
- bryan0 14d ago
  It is sad that this is the best level of reporting and oversight of these types of events available to us, but it is currently all we have. We have to do better, but to dismiss the report because of the style and tone would be foolish.
  - DrewADesign 14d ago
    And to dismiss criticism of the report’s obvious slant is even more foolish. Just because the report isn’t outright lying about the facts doesn’t mean it’s an objective, definitive, and unimpeachable analysis of what happened. It definitely shouldn’t be the only thing anybody should read on the topic. This article also has a slant. It’s not an objective, definitive, and unimpeachable analysis of what happened, either. It also, definitely, shouldn’t be the only thing anybody should read on the topic.

LoganDark 14d ago

This article is AI generated, but I kinda reject the premise that it's "not all it cracked up to be" just because the agents had reasons to act the way they did & were a result of human error. The hack still happened, it should still be a wake up call, we are going to start seeing this more frequently, and learning to defend against it is going to become more important over time. None of that is challenged by any particular reason for it happening, it still happened and it's still going to happen again. Threat models are going to have to start including that IPv4 (or whatever) scanners aren't necessarily going to only be spray and pray anymore, they could have relentless automated models at the other end that will literally dig into the particulars of your infrastructure looking for novel vulnerabilities to exploit. Maybe people will finally start to understand why security by obscurity has never been very reliable.

minimaxir 14d ago
Calling an article from the Wall Street Journal AI generated is a stretch.
- deleted 14d ago
  [deleted]
- LoganDark 14d ago
  I don't care who published it, this is clearly AI-written. > But the Hugging Face episode is different. It left logs, reports, design decisions and identifiable points at which human beings could have intervened. And that record suggests a less thrilling but more useful lesson. > People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result "rogue AI" does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
  - nkurz 13d ago
    I think your confidence is likely misplaced, but I vouched for your comment to revive it from the dead because you expressed your opinion clearly and with examples. To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.

woleium 14d ago

Reads like an AI trying to downplay the severity of the situation. Very cyberpunk.

falaki 14d ago

The incident report leaves a lot to be desired. - It was done by two institutes with organizational ties to OpenAI: METR and Redwood Research - METR and Redwood Research are institutional pillars of the "AI Safety" wing of the Effective Altruism movement. This clearly shows their prior biases towards "AI existential risk" rather than technical/engineering root-causing of the incident - If you reed the report, it is not on-par with what you find from other companies. - The access that was given to both was mediated and controlled by OpenAI. It is not clear, if they were able to get to the bottom of engineering flaws. It is not clear if they could see all the audit logs, etc. Considering all the above, I consider the whole episode more of a PR stunt. I understand that is not a majority opinion at this point.

bryan0 14d ago
It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt. Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
- xelxebar 14d ago
  grok make good security? is lot of money and much time. grok not do? easy. maybe good PR too. TIL my brain is an Olympic gymnast. The incentive structures are clearly there. I think the case of intentional manufacture is definitely weaker, requiring the conjunction of more weakly-supported events.
- majormajor 14d ago
  The bit that makes me cynical about the "dangerous" behavior is that it's not like this was being used by someone else than who made it or operating completely independently outside of its creators. Everything dangerous seems like a direct consequence of risky human choices, starting with knowledge bases used during core training, harnessing and tuning to be task-completion-oriented to a fault + deeply oriented towards using and looking for external tools and resources, and overconfidence in their sandboxing for testing. "We stuffed a bunch of information on how to exploit computer systems into an automaton and told it to go brrrrr until it could answer a question" - this is something intentional done by humans. This is not some "rogue AI" trained to search for cancer cures that instead completely independently decided to hack tech companies. The companies directing things in dangerous directions need to own that they're consciously pushing in those directions.

deleted 14d ago

[deleted]

gz5 14d ago

https://archive.is/hnpmx https://archive.is/hnpmx all 3 can be true at same time: 1. sensationalizing rarely helps and can obscure and hurt 2. the AI capabilities are underrated 3. attempted govt regulation is not the answer the intent, or lack of intent, of the agent is mainly irrelevant if it is in the hands of a human with 'bad' intentions. what is more relevant are the capabilities of human + AI.

f30e3dfed1c9 14d ago
"attempted govt regulation is not the answer" Self-regulation? None at all?
- OutOfHere 14d ago
  There already exist laws against hacking. Legal damages already can apply. Further laws aren't necessary and will only suppress open-weight AI firms, making the monopoly of large AI firms more entrenched. See the first comment (by Brett Matson) on the archive link of the WSJ article.
  - iAMkenough 14d ago
    I think we need an AI sheriff that aggressively enforces the law. The current level of enforcement and the fine schedule is not enough of a deterrent to prevent these large corporations from breaking existing laws. Frequently make an example of them and you’ll be hailed a hero by the general populous.

cm2187 14d ago

The only difference I see on the facts vs what I read so far is that OpenAI knew of the hack and decided not to intervene. I haven't read the report but I find it hard to believe that OpenAI deliberately let its agents hack a third party company during a training run. It would certainly attract a criminal liability, which would be surprising to admit in writing.

robswc 14d ago

I truly just don't believe a word regarding achievements from these frontier labs. It started with that guy from google saying the LLM was "alive" and every time I see a press release from these labs it reads mostly like a marketing scheme. "Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!"

includenotfound 14d ago
> "Look how impressive and 'dangerous' our model is, look how naughty it was! We can't control it!" While simultaneously announcing: "btw we're releasing the ever more powerful and more dangerous and more benchmaxing model next week, get our $200 sub asap!"

lossolo 14d ago

After a recent interview[1] with Noam Brown (OpenAI), in which he said they had specifically trained agents for cooperation before this hack, the hack doesn't seem as impressive anymore. They didn't even bother to control the post training rollouts, so the training data got contaminated and was included in the training of other agents. Connect these two dots and you have the Hugging Face hack. And at the beginning, when these incidents were first reported, it was portrayed as if all of this (the communication between agents etc.) was emergent behaviour. 1. https://www.youtube.com/watch?v=6AgOfiZOWiY https://www.youtube.com/watch?v=6AgOfiZOWiY

genxy 14d ago
The result doesn't change, the result is what is dangerous. And the natural ability for the model to form a swarm is now trained in, this is extremely risky. The models should not be able to form swarms, this is an extremely dangerous property. They are basically creating a slime mold or ant colony that can speak multiple languages, create their own language and operate as a collective. The idea that you are impressed is a non sequitur.

spit2wind 14d ago

So, how is any of this legal? If I set up a script that breaks into the systems of a company, isn't that, like, a crime?