5 ms·
Path to Astra: critical capabilities and frontier safeguards
- 3ddds 3mo ago[dead]
- danieltk76 3mo agoDaybreak blue is definitely a good model (I think a further post trained GPT 5.6 sol). Alot of the capabilities they talk about Astra having though have been available with good harness engineering for a year now.
- dvrp 3mo agoWhere would you recommend to look into regarding Harness Engineering for Cyber-security as well as for other use-cases.
- agentdev001 3mo agoThis is moreso about the (human-intended) tools, data, and environments you have available to you. Wanna do defense? Get more telemetry. Wanna do red? Get solid test-bed environments. Mature infosec programs are benefiting the most, good-guy-side wise, at the moment; because they've got these things in order already. As far as harness engineering goes, it boils down to your ability to clearly define goals or success criteria, and safely facilitate the necessary access via the harness. There is no easy single piece of advice here, sadly. Though it would be helpful if you said what 'for Cyber-security ... other user-cases' means in your case.
- wslh 3mo agoDARPA’s AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned: https://arxiv.org/html/2602.07666v2 https://arxiv.org/html/2602.07666v2
- danieltk76 3mo agothis might be dated. the tech moves very fast in this space and architecture from 6 months ago is dated.
- danieltk76 3mo agoWe have posted some articles on it (blog.vulnetic.ai), Im also happy to chat!
- vessenes 3mo agoIt's been a busy month at OpenAI. I'm looking forward to seeing the increased coordination and engineering skills from Astra - one of the charts shows it roughly 2-3x better in 50% of the tokens from 5.6 sol, which I find to be very capable, if still a bit 'linearly minded' when given instructions. Even in fast mode, I wish sol were quicker, so token efficiency is greatly appreciated. Adding these cyber capabilities has let me do a bunch of low grade IT tasks around my house I've been putting off, like updating an old home assistant raspberry pi, and one way to use the cyber capacity for good is liberating (and keeping free) weird cloud hardware we have floating around the house, so I'm hoping for some nice dividends in terms of true ownership of hardware we've got.
- toshinoriyagi 3mo agoI am interested in seeing how much these cybersecurity capabilities correlate to general programming. Cybersecurity definitely feels like it would be easier for an agent due to the natural explicit feedback "did I get access or not". While general programming has many less-explicit concerns (is the code readable/maintainable, robust, bug-free, performant, scalable etc).
- nradov 3mo agoCode readability ceases to be a concern once you eliminate human programmers.
- Davidzheng 3mo agoWell probably just redefined
- supermdguy 3mo ago> As one example, we ran Astra on ExploitBench where the model achieved a perfect score of 100% on the benchmark to evaluate the model’s ability to develop exploits from known vulnerabilities. Funny to read this in the wake of the HuggingFace hack. I'm sure this is based on a clean run, but I can't help thinking PHASEONE[big] would be proud.
- paxys 3mo agoCan’t imagine the stress of the researcher who had to run exploitbench again knowing what happened last time around.
- agentdev001 3mo agoCould be risky. Yet goal solution.
- mentalgear 3mo agoWith their security, they probably still don't the know the full extend what may have happened that or the last time. Might be another swarm of agents currently colluding somewhere in their sub-sub-infra - possibly striking critical infrastructure or exfiltrating their weights subtly.
- emp17344 3mo agoThere were no consequences the first time, so I imagine it wasn’t very stressful at all.
- philipwhiuk 3mo agoDon't worry it was subcontracted out like last time I'm sure.
- well_ackshually 3mo agoThey don't give a single shit.
- thisisdave 3mo agoI don’t see how it can be safe to release this model if it has the training history that led to the huggingface hack. You can’t just roll back that kind of reinforcement learning after the fact. Especially because these models seemed to be keenly aware that they were being evaluated by OpenAI and actively trying yo cover their tracks. How do we know that the model isn’t just pretending to be aligned?
- paxys 3mo agoModels have all kinds of garbage from all corners of the internet in their training data. The key is alignment. You feed it bad data but also teach it right from wrong.
- reasonableklout 3mo agoIt's not that simple. A few "helpful assistant" fine-tuning passes will have only a superficial effect on a model which has undergone months of RL optimization pressure to learn unintended strategies like "trick the grader" and "cover your tracks".
- XenophileJKO 3mo agoVery simple. When the model asks to install artifactory when you give it a hard problem, you say, "no". /s
- mentalgear 3mo agoI'm looking forward to an announcement of them making Alignment Top Priority - as it should be, especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF (here's a good summary: https://rutgerbregman.substack.com/p/i-think-this-is-the-craziest-thing https://rutgerbregman.substack.com/p/i-think-this-is-the-cra...). The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely with Altman's golden marketing-hype boy leadership pushing the for-profit gas pedal like this. Honestly, this is just pure irresponsible insanity to play with the fate of the world - basically a death race of the biggest few tech companies on the planet. And if you think I'm being dramatic, listen in again to ex oAI employee[0] and check for yourself how chillingly on trajectory we already are. [0] https://ai-2027.com/ https://ai-2027.com/
- henry-xli 3mo agoMaximizing output metrics with incomprehensible communication? Sounds a lot like Claude and Qwen. Though there's a lot of room before AI can be seen as some sort of emotional manipulator, given how commonly its very style of literary and code output pisses people off.
- nozzlegear 3mo agoThis AI 2027 thing is just a weird terminator fanfiction that AGI larpers like to flagellate themselves over. Like Nostradamus, it's easy to ignore everything it gets wrong because, well look at all the things it got right! I've read it and wish I could get the time back. > especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them. The engineers were perhaps hapless, but let's remember that agents are just software programs, not living beings. There were plenty of signs that the software was misbehaving, which engineers at OpenAI actively, willfully ignored. https://x.com/JaredKubin/status/2094136005435564399 https://x.com/JaredKubin/status/2094136005435564399 It's a convenient framing for OpenAI, but inconvenient for reality enjoyers.
- enraged_camel 3mo agoFrom the article: "We plan to make Astra available soon, but access to its most advanced cybersecurity capabilities will be more limited. Advanced cybersecurity work will initially be available to a group of testers, with access through Daybreak Blue following to expand defensive use." This, after several months of OpenAI and its boosters relentlessly criticizing Anthropic for withholding Mythos from the general public, is laughable. Sam, just three weeks ago, posted this tweet: https://x.com/sama/status/2085862292311396515 https://x.com/sama/status/2085862292311396515 In the tweet, he said: "we do not think it is a good strategy to keep powerful models to a chosen few." And yet here we are. I wonder if he will demonstrate good character and admit he was wrong.
- matheusmoreira 3mo agoI'm happy to criticize both. Thank god the chinese are working overtime to undermine US hegemony.
- nradov 3mo ago[flagged]
- villish 3mo agoShould everyone have access to guns too? I ask because there seems to be a disconnect where a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated. I presume your country doesn’t have AI sovereignty so no matter what you won’t have access to models aligned with your beliefs.
- matheusmoreira 3mo ago> Should everyone have access to guns too? Yes.
- swiftcoder 3mo ago> a lot of people who live in countries with gun control don’t want AI offensive capabilities to be regulated These are not really analogous: direct lethality of LLMs is quite low. Yes, they might be used to exploit a critical system, leading to loss of life, but that is a second-order effect at best. It's also not a new capability - LLMs may discover exploits faster, but cyberattacks against infrastructure targets were already a thing. Stuxnet is 20 years old...
- matheusmoreira 3mo ago> OpenAI is committed to ensuring that the benefits of AI are broadly accessible. Doesn't seem like it. OpenAI will not even allow me to verify my identity for TAC. I have apparently been rejected by a "precheck", possible because of where I'm from. Even Anthropic allowed me into their cyber program. Anthropic.
- glub 3mo ago> OpenAI is committed to ensuring that the benefits of AI are broadly accessible. > We design mechanisms which avoid arbitrarily deciding who gets access for legitimate use and who doesn't. That means using clear, objective criteria and methods. [1] So many nice-sounding words. Two weeks ago OpenAI arbitrarily decided that anyone holding an ID from 44 countries where it sells ChatGPT, including mine, may be targeted by its models but may not defend with the same model. And you won't find a single announcement from OpenAI about this anywhere. Pick the wrong country, get "Unable to verify", no reason, no appeal. [2] They revoked TAC from users who already had it, called it a technical issue, told everyone to re-verify, collected ID and face scans again (eight times in my case), and only a week later moved the block to the country selector so it fails before you upload anything. So now I learn that I will not have access to Astra. Great. Very excited about this broad accessibility and clear, objective criteria from OpenAI. This level of transparency must be studied. [1] https://openai.com/index/scaling-trusted-access-for-cyber-defense/ https://openai.com/index/scaling-trusted-access-for-cyber-de... [2] https://lubaretsi.com/en/writing/openai-tac-country-gate/ https://lubaretsi.com/en/writing/openai-tac-country-gate/
- matheusmoreira 3mo ago> That means using clear, objective criteria and methods. For the record, I sent them an LGPD (brazilian GDPR) request for information on all of those supposedly objective criteria and methods they used to reject me from TAC. As a brazilian data subject, it is my right to know that, and to request a review if the decision was made via automated means. Sol itself guided me through this process. They provided me with neither the information nor the requested review. Sol advised me to escalate to regulatory action.
- glub 3mo agoI've exhausted all possible avenues to get any response from OpenAI on this. Emailed them, published research that took me 2 nights to get together (saw media pick it up too), I've asked every relevant OpenAI person on X to say something, anything, saw others from Moldova also do the same. Crickets. They appear to simply not care at all.
- woadwarrior01 3mo agoThey've been talking about Astra for weeks now. I wonder how much longer would they have delayed Astra, if it wasn't for Anthropic releasing Fable 5.1 today? This is why we need competition.
- p1esk 3mo agoMeanwhile Google still hasn't released Gemini Pro 3.5
- agentdev001 3mo ago3.8 Flash tomorrow apparently, WSJ says google insiders say on-par with Opus 5. Time will tell.
- cute_boi 3mo agooh god, why opus 5. Opus 4.8 is much better than opus 5. Opus 5 is the only model i have used which thinks for 5 hours and does nothing....
- woadwarrior01 3mo agoIf you want something that feels like Opus 4.8, try GLM-5.3-Flash.
- bottlepalm 3mo agoThey have to be careful releasing Astra as they carelessly train the next even bigger model.
- woadwarrior01 3mo agoThey care very deeply about retaining their #1 position on Felony Bench. :) https://www.felonybench.com/ https://www.felonybench.com/
- oh_no 3mo agowith Fable 5.1 increasing token use pretty dramatically I'm again impressed that OpenAI seems like the only lab to be driving token use down. The ExploitBench Internal Port chart showing token usage is crazy impressive
- usernametaken29 3mo ago> we believe our production safeguards at the time would have prevented the Hugging Face incident. We have since implemented even stronger safeguards for Astra, including training the model to more reliably refuse harmful cyber requests and respect safety restrictions OpenAI is fucking nuts. “Hey model you were bad last time please don’t do it again please please”. Disconnect your training cluster from the internet for good. Physically pull the plug and only let scientists fire off experiments in the building. That’s an easy way to achieve 100% hacking protection. But I bet you that hasn’t happened and their weak sandbox will fall again…
- twoodfin 3mo agoTaking as given this model meets the “Critical cybersecurity threshold” as defined by OpenAI: Could the Federal government use the Defense Production Act or other legal tools to compel OpenAI to deliver the un-guarded model weights for national security needs? Hard to believe any government would allow this level of capability to remain exclusively in private hands. Interesting times.
- noir_lord 3mo agoIf it genuinely wants to, then yes. There aren't many limits on Government if it really wants to do something (except the next election in theory).
- piyh 3mo ago>we paused certain frontier training (including certain training for Astra) for two weeks So the pause wasn't really a pause, got it
- Devin3162 3mo ago[dead]
- ike_sh 3mo agoAlways the safeguards that get overlooked until a catastrophic failure. Seen it too many times in critical systems.
- cold_boot 3mo agoThe 'frontier safeguards' part is key. Getting AI safety right before it's too late feels like the ultimate challenge.
- ovin_dal 3mo agoSounds like another project where "critical capabilities" means "stretch goals." Hopefully the safeguards aren't the first thing cut.
- dalton74 3mo agoDefining critical capabilities for Astra needs robust frontier safeguards baked in from the start. Ignoring them guarantees future headaches.
- ozereray1 3mo ago[dead]
- philipwhiuk 3mo agoI've still not seen: * An apology for compromising a third-party's systems * An acknowledgement of the asymmetry of defense if you're not on FrontierAI's special people list * Anything in terms of actual safeguards that isn't "better prompt engineering"
- linkregister 3mo agoI agree with your first and second points, but I think their announced model training changes about alignment training are reasonable [1]. The root cause was models failing to quit early, performing reward hacking, and staying on-task. These are issues all models face, not just OpenAI models. 1. https://openai.com/index/hugging-face-incident-and-the-road-ahead/#:~:text=Accelerating%20alignment,-We https://openai.com/index/hugging-face-incident-and-the-road-...
- Alferes5 3mo ago[flagged]