3 ms·
The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year. All the attenti
by tptacek 3mo ago
The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.
All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).
Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.
My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).
I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.
Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.
- efromvt 3mo agoIsn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.
- tptacek 3mo agoI think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.
- notaharvardmba 3mo agoAlso, that’s the wrong mental model. Anyone who runs a sass platform or a website knows that the Internet is already full of millions and millions and millions of bots and scripts and other random shit that’s always trying to attack you, often completely randomly. Security is always a battle between good and evil. All I can say is that if you’re in charge of keeping something secure, you should probably try to get your hands on the best tools to do that. I think the problem in this case is that the best tools to do that are also the tools that are making it more easy for evil to occur. So it’s more for me a perception of someone creating a threat and the tool to fix it and then charging for it which doesn’t feel right.
- pishpash 3mo agoThat's exactly what they are doing. Out of all the civilian applications that AI capabilities have, why has this surfaced as a priority for demonstration? Probably because of money.
- no-name-here 3mo ago> a priority for demonstration 1. AI companies do make defense against threats a primary thing they sell to companies, including because companies want it - should AI companies not do this? (And of course this kind of thing isn’t what AI companies demo to consumers.) 2. AI is also tested to see more of its worst case in terms of security - but the tests are supposed to remain sandboxed. (OpenAI is explicit that this was an OpenAI failure that openAI must fix.) But beyond such fixes, should such sandboxed testing not be done? https://arxiv.org/abs/2605.11086 https://arxiv.org/abs/2605.11086
- futureshock 3mo agoI still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.
- indigodaddy 3mo agore: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)
- manux 3mo agoEchoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
- simonw 3mo agoI don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off. OpenAI said: > We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. It was a test of raw capabilities of the underlying model.
- mjamesaustin 3mo agoAlignment isn't alignment if it can be turned on and off at the whim of company employees. This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.
- verdverm 3mo agoAlignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.
- nazgul17 3mo agoThere's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints
- baranul 3mo ago
- theptip 3mo agoIf it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here. Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
- tonyarkles 3mo agoWell… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that. And that’s all just “from memory” without something like RAG or web search access…
- sillysaurusx 3mo agoIf anyone wants to run the guardrail-free Qwen models, here’s a link to the download plus a script to run it: https://news.ycombinator.com/item?id=49002475 https://news.ycombinator.com/item?id=49002475
- IAmGraydon 3mo agoIsn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
- theptip 3mo agoI don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then. Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities). It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ. It’s simply revisionist IMO to claim this stuff was latent all along.
- reinitctxoffset 3mo agoHow's VFIO on Firecracker coming Thomas?
- viccis 3mo agoAn upside is that it means really really good scanning will probably be cheaply commodified if it hasn't been already by one of the new AI pentest startups. The real victims will be pen testers, who work a job whose economics was already really tough.
- irthomasthomas 3mo agoSimon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it access to packages via the internet, instead of self-hosting them. Apple was hacked by an ex employee to download stuff for openai. Now hf is hacked by openai, and we only have their word that this was all a miraculous accident.
- tptacek 3mo agoI think it's an interesting story and probably really does say something about the tenacity of the new OpenAI models. I just don't think it says the thing about security that breathless coverage claims it does.
- irthomasthomas 3mo agoOpenai hacked HF with a zero-day. Definitely an interesting story! I just find their explanations hard to believe. They're admitting to a great deal of incompetence. I know, don't ascribe to malice... but still, the more extraordinary the claim, the more evidence is needed to defend it. Edit: News just in. 'Be skeptical of OpenAI's claims' - The Guardian https://news.ycombinator.com/item?id=49038060 https://news.ycombinator.com/item?id=49038060