8 ms·

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-c

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.

ajyoon 2mo ago
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
- artrockalter 2mo ago
  I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
  - ajyoon 2mo ago
    Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
    - verdverm 2mo ago
      Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
- tekacs 2mo ago
  This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders. For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
  - dwaltrip 2mo ago
    The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once". You are suggesting this isn't correct? > a defender gets to pick the surface area What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
    - sudosysgen 2mo ago
      > "defenders have to be right 100% of the time, while attackers only have to be right once" If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
      - pastel8739 2mo ago
        Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
        sudosysgen 2mo ago
        It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
- heyjstn 2mo ago
  are you working at anthropic? you're literally repeat what that freaking Dario say everyday
alienbaby 2mo ago
? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.
- akersten 2mo ago
  huggingface asked the frontier models to help them analyze the attack and lock down their systems the frontier models refused because their cyber detector went off they had to use GLM 5.2 instead
  - usef- 2mo ago
    Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
    - marcus_holmes 2mo ago
      One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
  - paxys 2mo ago
    They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.
deleted 2mo ago
[deleted]
paxys 2mo ago
> Only by using the open source GLM-5.2 were they able to survive an attack They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing. If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
varenc 2mo ago
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire. Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
- Majromax 2mo ago
  > The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link. In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration. I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers. > Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie) In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
vultour 2mo ago
I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.

ajyoon 2mo ago

Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?

artrockalter 2mo ago
I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
- ajyoon 2mo ago
  Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
  - verdverm 2mo ago
    Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
tekacs 2mo ago
This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders. For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
- dwaltrip 2mo ago
  The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once". You are suggesting this isn't correct? > a defender gets to pick the surface area What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
  - sudosysgen 2mo ago
    > "defenders have to be right 100% of the time, while attackers only have to be right once" If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
    - pastel8739 2mo ago
      Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
      - sudosysgen 2mo ago
        It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
heyjstn 2mo ago
are you working at anthropic? you're literally repeat what that freaking Dario say everyday

alienbaby 2mo ago

? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.

akersten 2mo ago
huggingface asked the frontier models to help them analyze the attack and lock down their systems the frontier models refused because their cyber detector went off they had to use GLM 5.2 instead
- usef- 2mo ago
  Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
  - marcus_holmes 2mo ago
    One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
- paxys 2mo ago
  They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.

deleted 2mo ago

[deleted]

paxys 2mo ago

> Only by using the open source GLM-5.2 were they able to survive an attack They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing. If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.

varenc 2mo ago

You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire. Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)

Majromax 2mo ago
> The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link. In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration. I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers. > Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie) In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.

vultour 2mo ago

I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.