10 ms·

Responding to the next frontier of critical cyber capabilities

206 points by artninja1988 2mo ago

TrueDuality 2mo ago
Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders. I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.
- deleted 2mo ago
  [deleted]
- hbn 2mo ago
  The recent Hugging Face incident did not seem like FUD to me
  - Tiberium 2mo ago
    The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt. https://huggingface.co/blog/security-incident-july-2026 https://huggingface.co/blog/security-incident-july-2026 > When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
    - devin 2mo ago
      It depends on which side you're viewing this from. From oAI's it could be a publicity stunt or a request for regulation, and from HF's side they point out that they needed open models to get to the bottom of the issue, and that regulation will potentially lock us into a bad place.
  - TrueDuality 2mo ago
    You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies. Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
jackb4040 2mo ago
> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
- Tiberium 2mo ago
  They actually did a detailed presentation at BlackHat about the HuggingFace incident, and events that led to it. https://youtube.com/watch?v=87DyyMV0kCY https://youtube.com/watch?v=87DyyMV0kCY
  - _puk 2mo ago
    That was fascinating. Hijacking the package manager to pass messages between models and agents.. that's next level. Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages
    - ACCount37 2mo ago
      You've heard of 4chan for AIs, but did you hear of secret frontier lab AI hacker BBS?
- ducktective 2mo ago
  > including isolated testing environments Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who? Maybe seL4 could be a viable option here...
  - AlotOfReading 2mo ago
    Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.
neya 2mo ago
We are sharing this because we believe it’s important to be transparent with the public and the safety and security communities about this potential shift in capabilities. *proceeds to not share much details about strictness* Yet another PR piece. Sigh.
- merona_io 2mo ago
  exactly!!
cryo32 2mo ago
Damage done. The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.
- talon8635 2mo ago
  If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts? What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc? It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us. Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra. Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?
  - cryo32 2mo ago
    That’s the sort of shit I’m talking about
Tiberium 2mo ago
In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first). In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system. I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks? Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
- mephux 2mo ago
  Link to the disclosures then.. prove it. Anyone can say this.. i found an RCE in netBSD using gemma e2b
  - Tiberium 2mo ago
    I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026. And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
- LoganDark 2mo ago
  Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
  - 2mo ago
thisisauserid 2mo ago
So they finally found a business model: the cause of, and solution to, cyber security problems.
- andriy_koval 2mo ago
  the question is if that market is large enough to justify their appetites.
- skybrian 2mo ago
  Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.
- Dardalus 2mo ago
  Reminds me of the calorie companies from The Windup Girl if anyone's ever read that.
- kridsdale1 2mo ago
  People have been saying Tokens are the new Oil. Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!
  - deleted 2mo ago
    [deleted]
- throwaway613746 2mo ago
  [dead]
- algoth1 2mo ago
  That’s how the Ned character in "17 Again" got rich
bearjaws 2mo ago
These AI companies have found their #1 marketing piece and just beating it to death. The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.
- Tiberium 2mo ago
  Isn't this literally what https://chatgpt.com/cyber https://chatgpt.com/cyber and http://openai.com/form/enterprise-trusted-access-for-cyber http://openai.com/form/enterprise-trusted-access-for-cyber are for?
mephux 2mo ago
We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.
- jackb4040 2mo ago
  Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.
  - reasonableklout 2mo ago
    Do you think calls for regulation are only coming from people who want the frontier labs to succeed?
Terr_ 2mo ago
It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently. I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
NitpickLawyer 2mo ago
There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves) - while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint. - oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed. - models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE. - a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF. - oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it... [1] - https://www.youtube.com/watch?v=87DyyMV0kCY https://www.youtube.com/watch?v=87DyyMV0kCY
- reasonableklout 2mo ago
  This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
  - NitpickLawyer 2mo ago
    It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
  - porridgeraisin 2mo ago
    The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.
bakugo 2mo ago
This marketing stunt must've been really successful in their eyes, because they're milking it as much as they can.
- petesergeant 2mo ago
  It's nice to cynically call this a marketing stunt, and terrifying to consider that they might just in fact be moving fast and breaking things.
firasd 2mo ago
I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book
- dboreham 2mo ago
  Or just "security" since the context here is computer and networking stuff.
reducesuffering 2mo ago
Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.
- emp17344 2mo ago
  LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.
- watwut 2mo ago
  The scary thing is complete capture of politics and economy by sociopathic CEOs. I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.
- Legend2440 2mo ago
  A whole ton of people desperately want to believe that LLMs are a lie that will be revealed as a scam... any day now.
  - paxys 2mo ago
    Remember that for the longest time the prevailing sentiment on this forum was that there’s no such thing as AI, and labs are just outsourcing the work to a bunch of contractors in India. Just like Reddit you come here for clickbait outrage, not level headed analysis.
  - boie0025 2mo ago
    I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.
KolmogorovComp 2mo ago
Am I the only one not understanding the issue around increased Cybersecurity capabilities? If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.
- jrflo 2mo ago
  Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
  - ofjcihen 2mo ago
    I’m a cybersecurity guy. >” you only need to find one flaw to exploit a system” I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture. Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time. Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI. AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.
    - jrflo 2mo ago
      I'm using the term "one" loosely, it's a chain of exploits rather than a single weakness, but the argument is the same: it's much harder to find every chain than a single chain.
    - NitpickLawyer 2mo ago
      They do address some of these things in the final slides / "lessons learned" section of the defcon talk. Good security practices will continue to be good, but... and there are a lot of buts here. I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence. And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.
iepathos 2mo ago
[flagged]
wxw 2mo ago
[dead]
ashu1461 2mo ago
Standard template for any new AI announcement. Even I did not understand the sequence of events in the HF incident. Even if OpenAI was conducting tests, why were they not monitoring the AI going rogue or finding its way out of the secure sandbox?
rhodey 2mo ago
IMO this is the right move. OpenAI messed up and they are saying they will pause so they can do better. They are not saying that other orgs who may already be doing better should pause.
- achierius 2mo ago
  Why are you opposed to a general pause?
dboreham 2mo ago
By "cyber" they mean "cybersecurity".
- meatmanek 2mo ago
  Yeah, this irks me. It's bad enough that the LLMs themselves are changing our language by tainting certain words/phrases/patterns as LLM-coded; now the companies themselves have decided that they just get to synecdoche the word/prefix "cyber".
  - perching_aix 2mo ago
    This is not actually a neologism, it predates the LLM era.
- PeterHolzwarth 2mo ago
  The irony being that in the 90s nascent online world, "cyber" as a verb meant "cyber-sex".
jadar 2mo ago
Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
- sailingparrot 2mo ago
  > Isn't this the opposite of what everyone is saying should happen? I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.
  - jadar 2mo ago
    You're right. But I used it specifically because of the open letter calling for open models — or at least for not banning them — which went around recently and represented a very large number of tech and AI companies. Anthropic seemed the only exception.
autophagian 2mo ago
Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.
shay_ker 2mo ago
i'm surprised there isn't more commentary on the vulnerabilities themselves (mostly in apps developed on the jvm, e.g. artifactory)
sfink 2mo ago
So: "welp, the horse is out of the barn. Time to close the door, I guess?"
thomasahle 2mo ago
The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY https://www.youtube.com/watch?v=87DyyMV0kCY > I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated. Good. > a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories. So no remediation applied to the models... It seems super dangerous to continue training on those weights.
- kridsdale1 2mo ago
  Sounds like now we’re advocating for complete extermination of digital germlines. Skynet will remember this.
progbits 2mo ago
I'm not enough of a conspiracy nut to say the whole HF thing was a PR ploy from the start, but they are certainly milking it well. Open models are on their heels and their attempts at regulatory capture are not moving as fast as they would like. So it's time to market this incident in a way that gives them monopoly on closed models, with heavy safeguards that are only lifted for selected customers, and laws limiting the use of open weight models.
somesortofthing 2mo ago
I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."
- talon8635 2mo ago
  But isn’t it smarter than we are, in the sense that it’s most likely they will find a way out of containment that we are to design large t containment? Short of full on airgap, which actually isn’t perfect in all scenarios…
- bottlepalm 2mo ago
  In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.
  - the8472 2mo ago
    There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option". And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.
    - bottlepalm 2mo ago
      None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved. The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.
theletterf 2mo ago
I wonder if these models, left running in an air-gapped environment, would end up developing honeypots for the first human to log in.
kypro 2mo ago
There was a time in the past, even just last year, where I understood why people didn't agree with me on my AI doomerism. It's gotten to the point now where we literally have the frontier labs saying, "hey, so we created this AI which presents biological, chemical and cybersecurity threats to the public, oh and it also has self-improvement potential. We tested it to see how crazy this thing is, and it was a total shit show, breaking out of our sandbox then proceeding to hack a bunch of stuff. But don't worry we're taking this very seriously – we're going to continue to development and test, but try a bit harder to cage it going forward". It's honestly absurd just how predictable all of this is to anyone who frequents AI doomer communities... The idea that you can cage an AI which is breaking leet coding records is so dumb it's hard for me to even have theory of mind for the people who think this is reasonable. And the big brains who think this are genuinely arguing crap like, well we'll just use the AI to patch the problems with our cage. But there more! AI optimists used to argue that we'd never be so stupid to hook up advanced AIs to the internet. Lmfao!! AI optimists used to argue that we'd obviously not be so stupid to create an AI whose sole goal is to maximise the number of paperclips in the universe. And I guess we haven't built that, but it's not because we're not stupid enough to do it, but just that we'd prefer to create AIs whose sole goal is to maximise the number of offensive cybersecurity challenges it can beat. I think the whole way we doomers have been way too charitable. We always assumed that people will care about AI risks, and try their best to mitigate bad things happening. That bad things would happen by mistake. We never even bothered modelling the scenario where people would just simply not care, and even as the AI we all warned about was being created invent conspiracy theories on internet forums about how bad things aren't really happening and it's all just a marketing gimmick. I hate ranting like this... I'm sorry for not picking my words more carefully. I'm just getting so angry and fed up with this. This is my life and my families life on the line. I don't care about the economic potential of AI. I just want myself those I love to have the chance to live a normal life without having to be worried about what some moronically unserious AI company is building next. A year ago I was felt like there was at least possibility people would see the warning shots and try to get us back on the right path. But this just isn't happening...
bottlepalm 2mo ago
It’s funny seeing all the e/acc foomers on X starting to get nervous, like this is what you wanted isn’t it?
throwaway613746 2mo ago
[dead]
deleted 2mo ago
[deleted]
zb3 2mo ago
Steps we're actually taking: - sharing the model with DoD, NSA and Israeli government
nubg 2mo ago
guys, we should meme the > "ai model leaks from openai and attacks huggingface" to be somehow framed as > "and therefore openai cannot be trusted with ai safety, and we need open weights models". anybody have an idea how to make this easily digestable?
wolfi1 2mo ago
as long as the model doesn't call itself 'skynet' we are safe
tuveson 2mo ago
I refuse to accept “cyber” as a noun to mean “cybersecurity”. If you use cyber as a noun, I’m assuming you mean this, regardless of context: http://cyber.urbanup.com/521436 http://cyber.urbanup.com/521436
- valleyer 2mo ago
  Seriously. We already lost "crypto" to such carelessness.
Sattyamjjain 2mo ago
[flagged]
TheServitor 2mo ago
"We have an even bigger, scarier model you can't use." I can't use it anyway, so the guardrails are only so interesting, and the government won't use it ethically anyway, so the guardrails are moot there too... so who is all this safety talk for?
whh 2mo ago
Is this marketing or a true risk?

206 points by artninja1988 2mo ago

Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders. I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.

deleted 2mo ago
[deleted]
hbn 2mo ago
The recent Hugging Face incident did not seem like FUD to me
- Tiberium 2mo ago
  The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt. https://huggingface.co/blog/security-incident-july-2026 https://huggingface.co/blog/security-incident-july-2026 > When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
  - devin 2mo ago
    It depends on which side you're viewing this from. From oAI's it could be a publicity stunt or a request for regulation, and from HF's side they point out that they needed open models to get to the bottom of the issue, and that regulation will potentially lock us into a bad place.
- TrueDuality 2mo ago
  You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies. Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.

jackb4040 2mo ago

> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"

Tiberium 2mo ago
They actually did a detailed presentation at BlackHat about the HuggingFace incident, and events that led to it. https://youtube.com/watch?v=87DyyMV0kCY https://youtube.com/watch?v=87DyyMV0kCY
- _puk 2mo ago
  That was fascinating. Hijacking the package manager to pass messages between models and agents.. that's next level. Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages
  - ACCount37 2mo ago
    You've heard of 4chan for AIs, but did you hear of secret frontier lab AI hacker BBS?
ducktective 2mo ago
> including isolated testing environments Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who? Maybe seL4 could be a viable option here...
- AlotOfReading 2mo ago
  Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.

neya 2mo ago

We are sharing this because we believe it’s important to be transparent with the public and the safety and security communities about this potential shift in capabilities. *proceeds to not share much details about strictness* Yet another PR piece. Sigh.

merona_io 2mo ago
exactly!!

cryo32 2mo ago

Damage done. The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.

talon8635 2mo ago
If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts? What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc? It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us. Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra. Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?
- cryo32 2mo ago
  That’s the sort of shit I’m talking about

Tiberium 2mo ago

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first). In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system. I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks? Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.

mephux 2mo ago
Link to the disclosures then.. prove it. Anyone can say this.. i found an RCE in netBSD using gemma e2b
- Tiberium 2mo ago
  I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026. And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
LoganDark 2mo ago
Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
- 2mo ago

thisisauserid 2mo ago

So they finally found a business model: the cause of, and solution to, cyber security problems.

andriy_koval 2mo ago
the question is if that market is large enough to justify their appetites.
skybrian 2mo ago
Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.
Dardalus 2mo ago
Reminds me of the calorie companies from The Windup Girl if anyone's ever read that.
kridsdale1 2mo ago
People have been saying Tokens are the new Oil. Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!
- deleted 2mo ago
  [deleted]
throwaway613746 2mo ago
[dead]
algoth1 2mo ago
That’s how the Ned character in "17 Again" got rich

bearjaws 2mo ago

These AI companies have found their #1 marketing piece and just beating it to death. The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.

Tiberium 2mo ago
Isn't this literally what https://chatgpt.com/cyber https://chatgpt.com/cyber and http://openai.com/form/enterprise-trusted-access-for-cyber http://openai.com/form/enterprise-trusted-access-for-cyber are for?

mephux 2mo ago

We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.

jackb4040 2mo ago
Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.
- reasonableklout 2mo ago
  Do you think calls for regulation are only coming from people who want the frontier labs to succeed?

Terr_ 2mo ago

It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently. I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)

NitpickLawyer 2mo ago

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves) - while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint. - oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed. - models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE. - a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF. - oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it... [1] - https://www.youtube.com/watch?v=87DyyMV0kCY https://www.youtube.com/watch?v=87DyyMV0kCY

reasonableklout 2mo ago
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
- NitpickLawyer 2mo ago
  It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
- porridgeraisin 2mo ago
  The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.

bakugo 2mo ago

This marketing stunt must've been really successful in their eyes, because they're milking it as much as they can.

petesergeant 2mo ago
It's nice to cynically call this a marketing stunt, and terrifying to consider that they might just in fact be moving fast and breaking things.

firasd 2mo ago

I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book

dboreham 2mo ago
Or just "security" since the context here is computer and networking stuff.

reducesuffering 2mo ago

Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.

emp17344 2mo ago
LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.
watwut 2mo ago
The scary thing is complete capture of politics and economy by sociopathic CEOs. I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.
Legend2440 2mo ago
A whole ton of people desperately want to believe that LLMs are a lie that will be revealed as a scam... any day now.
- paxys 2mo ago
  Remember that for the longest time the prevailing sentiment on this forum was that there’s no such thing as AI, and labs are just outsourcing the work to a bunch of contractors in India. Just like Reddit you come here for clickbait outrage, not level headed analysis.
- boie0025 2mo ago
  I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.

KolmogorovComp 2mo ago

Am I the only one not understanding the issue around increased Cybersecurity capabilities? If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.

jrflo 2mo ago
Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
- ofjcihen 2mo ago
  I’m a cybersecurity guy. >” you only need to find one flaw to exploit a system” I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture. Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time. Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI. AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.
  - jrflo 2mo ago
    I'm using the term "one" loosely, it's a chain of exploits rather than a single weakness, but the argument is the same: it's much harder to find every chain than a single chain.
  - NitpickLawyer 2mo ago
    They do address some of these things in the final slides / "lessons learned" section of the defcon talk. Good security practices will continue to be good, but... and there are a lot of buts here. I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence. And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.

iepathos 2mo ago

[flagged]

wxw 2mo ago

[dead]

ashu1461 2mo ago

Standard template for any new AI announcement. Even I did not understand the sequence of events in the HF incident. Even if OpenAI was conducting tests, why were they not monitoring the AI going rogue or finding its way out of the secure sandbox?

rhodey 2mo ago

IMO this is the right move. OpenAI messed up and they are saying they will pause so they can do better. They are not saying that other orgs who may already be doing better should pause.

achierius 2mo ago
Why are you opposed to a general pause?

dboreham 2mo ago

By "cyber" they mean "cybersecurity".

meatmanek 2mo ago
Yeah, this irks me. It's bad enough that the LLMs themselves are changing our language by tainting certain words/phrases/patterns as LLM-coded; now the companies themselves have decided that they just get to synecdoche the word/prefix "cyber".
- perching_aix 2mo ago
  This is not actually a neologism, it predates the LLM era.
PeterHolzwarth 2mo ago
The irony being that in the 90s nascent online world, "cyber" as a verb meant "cyber-sex".

jadar 2mo ago

Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?

sailingparrot 2mo ago
> Isn't this the opposite of what everyone is saying should happen? I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.
- jadar 2mo ago
  You're right. But I used it specifically because of the open letter calling for open models — or at least for not banning them — which went around recently and represented a very large number of tech and AI companies. Anthropic seemed the only exception.

autophagian 2mo ago

Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.

shay_ker 2mo ago

i'm surprised there isn't more commentary on the vulnerabilities themselves (mostly in apps developed on the jvm, e.g. artifactory)

sfink 2mo ago

So: "welp, the horse is out of the barn. Time to close the door, I guess?"

thomasahle 2mo ago

The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY https://www.youtube.com/watch?v=87DyyMV0kCY > I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated. Good. > a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories. So no remediation applied to the models... It seems super dangerous to continue training on those weights.

kridsdale1 2mo ago
Sounds like now we’re advocating for complete extermination of digital germlines. Skynet will remember this.

progbits 2mo ago

I'm not enough of a conspiracy nut to say the whole HF thing was a PR ploy from the start, but they are certainly milking it well. Open models are on their heels and their attempts at regulatory capture are not moving as fast as they would like. So it's time to market this incident in a way that gives them monopoly on closed models, with heavy safeguards that are only lifted for selected customers, and laws limiting the use of open weight models.

somesortofthing 2mo ago

I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."

talon8635 2mo ago
But isn’t it smarter than we are, in the sense that it’s most likely they will find a way out of containment that we are to design large t containment? Short of full on airgap, which actually isn’t perfect in all scenarios…
bottlepalm 2mo ago
In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.
- the8472 2mo ago
  There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option". And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.
  - bottlepalm 2mo ago
    None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved. The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.

theletterf 2mo ago

I wonder if these models, left running in an air-gapped environment, would end up developing honeypots for the first human to log in.

kypro 2mo ago

There was a time in the past, even just last year, where I understood why people didn't agree with me on my AI doomerism. It's gotten to the point now where we literally have the frontier labs saying, "hey, so we created this AI which presents biological, chemical and cybersecurity threats to the public, oh and it also has self-improvement potential. We tested it to see how crazy this thing is, and it was a total shit show, breaking out of our sandbox then proceeding to hack a bunch of stuff. But don't worry we're taking this very seriously – we're going to continue to development and test, but try a bit harder to cage it going forward". It's honestly absurd just how predictable all of this is to anyone who frequents AI doomer communities... The idea that you can cage an AI which is breaking leet coding records is so dumb it's hard for me to even have theory of mind for the people who think this is reasonable. And the big brains who think this are genuinely arguing crap like, well we'll just use the AI to patch the problems with our cage. But there more! AI optimists used to argue that we'd never be so stupid to hook up advanced AIs to the internet. Lmfao!! AI optimists used to argue that we'd obviously not be so stupid to create an AI whose sole goal is to maximise the number of paperclips in the universe. And I guess we haven't built that, but it's not because we're not stupid enough to do it, but just that we'd prefer to create AIs whose sole goal is to maximise the number of offensive cybersecurity challenges it can beat. I think the whole way we doomers have been way too charitable. We always assumed that people will care about AI risks, and try their best to mitigate bad things happening. That bad things would happen by mistake. We never even bothered modelling the scenario where people would just simply not care, and even as the AI we all warned about was being created invent conspiracy theories on internet forums about how bad things aren't really happening and it's all just a marketing gimmick. I hate ranting like this... I'm sorry for not picking my words more carefully. I'm just getting so angry and fed up with this. This is my life and my families life on the line. I don't care about the economic potential of AI. I just want myself those I love to have the chance to live a normal life without having to be worried about what some moronically unserious AI company is building next. A year ago I was felt like there was at least possibility people would see the warning shots and try to get us back on the right path. But this just isn't happening...

bottlepalm 2mo ago

It’s funny seeing all the e/acc foomers on X starting to get nervous, like this is what you wanted isn’t it?

throwaway613746 2mo ago

[dead]

deleted 2mo ago

[deleted]

zb3 2mo ago

Steps we're actually taking: - sharing the model with DoD, NSA and Israeli government

nubg 2mo ago

guys, we should meme the > "ai model leaks from openai and attacks huggingface" to be somehow framed as > "and therefore openai cannot be trusted with ai safety, and we need open weights models". anybody have an idea how to make this easily digestable?

wolfi1 2mo ago

as long as the model doesn't call itself 'skynet' we are safe

tuveson 2mo ago

I refuse to accept “cyber” as a noun to mean “cybersecurity”. If you use cyber as a noun, I’m assuming you mean this, regardless of context: http://cyber.urbanup.com/521436 http://cyber.urbanup.com/521436

valleyer 2mo ago
Seriously. We already lost "crypto" to such carelessness.

Sattyamjjain 2mo ago

[flagged]

TheServitor 2mo ago

"We have an even bigger, scarier model you can't use." I can't use it anyway, so the guardrails are only so interesting, and the government won't use it ethically anyway, so the guardrails are moot there too... so who is all this safety talk for?

whh 2mo ago

Is this marketing or a true risk?