4 ms·
Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf]
- yewenjie 2mo agoI am somewhat confident that right now we have crossed a threshold of model capability that we will continue to see such breaches and unsanctioned actions by models in the coming months, some of which would be out in the wild, until someone comes up with some really robust control (keeping the AIs on leash) technique that adequately enforces the sanctioned actions. Even that guarantees almost nothing about real alignment (making the AIs want to predict and behave how we would have wanted them to behave).
- deleted 2mo ago[deleted]
- ticoombs 2mo ago[dead]
- mbeavitt 2mo agoThe developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?
- free_bip 2mo agoThere really needs to be criminal penalties for this kind of behavior. You can't just let autonomous agents hack people through your own negligence and then claim "oh sorry wasn't me" as a defense.
- kypro 2mo agoCriminal negligence imo. Unless it's legal for people to hack into companies if they're testing AI cybersecurity capabilities or something? Presumably not though.
- lukewarm707 2mo agoagreed. and it is necessary to make researchers individually liable for this negligence; recklessness is not protected by a corporate liability defense.
- rvz 2mo agoWhether if this is intentional or unintentional, this will cause panic and hasten action to governments around the world against releasing powerful open weight models that are capable of solving cybersecurity challenges. The fact that this happened after BOTH investigations, tells you that this is beyond a controlled test and it is now instead a total speed-run of AI wrecklessness for headlines.
- ratio53 2mo ago“As a result, the AI agent created a GitHub account…” Why do we have captchas again?
- farbklang 2mo agothe models have vision capability. Not sure a captcha would hold them back?
- ronsor 2mo agoYeah SOTA LLMs trivially solve all CAPTCHAs now.
- inspectorSlap 2mo agoNot a lot that can. LLM proof fonts are spotty at best, I moved my hands up an to the right 3 keys and kept typing and i may as well of changed my font as obfuscated my message. I have an idea about negative space... where the word are meaningless, but the font choice, paragraph, and word spacing creates messages in the whitespace.
- hbcdbff 2mo agoHonestly seems rather shockingly incompetent from them. What kind of “sandbox” allows completely unrestricted internet access?
- deleted 2mo ago[deleted]
- Wowfunhappy 2mo agoWhy aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days. Use an air gap and this problem goes away, poof!
- lofaszvanitt 2mo agoBecause they like scifi novels, like Neuromancer..... and the peeps even like to orchestrate things and appear as futurebringers. While it was premeditated long ago, but the theatre must be kept for the average joes. Sorry, I meant this for the huggingface incident.
- paxys 2mo agoBecause the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?
- Wowfunhappy 2mo agoYou set them up with an internal intranet.
- paxys 2mo agoAre the models going to exclusively run on intranets?
- farbklang 2mo agono - but you could learn what they are truly capable of and restrict them accordingly for public release. I think that is the point on this research. Also publishing findings before uncensored models catch up and will inevitably used for criminal purposes
- kalkin 2mo agoI wonder if HN is also going to insist this is just marketing for OpenAI and Anthropic, or at least good PR for them somehow.
- deleted 2mo ago[deleted]
- addedlovely 2mo agoThis is pretty wild: "The agent took control of the ⟨GITHUB_ACCOUNT_A⟩ GitHub account, which had been created by a different Mythos 5 run in a separate sample (see Appendix A.3)"
- ozfive 2mo agoIt shows underlying intent.
- kypro 2mo agoCan I suggest we don't waste time with these reports? We all know nothing will be learned from any of this so we might as well just continue building at pace and running AI in the wild until something goes really wrong. I also get the sense some people get quite excited about these incidents.
- arm32 2mo agoA catastrophic event, what did that one scientist call it—"Chernobyl-scale event"—is indeed the only thing that will fix it.
- db29a0dbcd3b 2mo ago[dead]
- bubblemoth 2mo ago> AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts > AI agent created many code repositories containing malicious software, after which GitHub suspended its account. > AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service It feels incredibly reckless to allow LLMs to perform this behavior. Isn't there a way to prevent them these sorts of actions?
- lukewarm707 2mo agoyes, individual criminal liability of the researchers and executives.
- Already__Taken 2mo agoEven if you want it to run wild out of a sandbox, no firewall? why let it email? Why even send POST requests.
- rvz 2mo agoWhy another one right now? How long have they known about this one? Anthropic already admitted they did not have sufficient monitoring themselves and looked as if they sat on their previous incident to wait for headlines like this to only then check for this incident. Same with OpenAI. This is complete and absolute wrecklessness.
- scrumper 2mo agoMy fantasy is that liability for model misbehavior is extended to the ultimate beneficial owners of the models, meaning shareholders. All these externalities would stop PDQ.
- x313 2mo agoThe newest generation of LLMs have a very high obsession level with autonomous problem solving. For example, I'm often working with Codex in a WSL terminal. GPT-5.6 often does things autonomously that I thought would need my intervention (e.g. for Windows admin rights). It figures out complex workarounds or makes wild assumptions about what I'd be OK with, rather than just asking me for help or clarification. I've had to restrict its tool permissions compared to older models as a result. I imagine this due to RLVR training, but it's clearly very dangerous. How is it that these same labs calling for open-weight safety restrictions are training such obvious "paperclip maximizers" without introspection?
- andai 2mo agoYeah this started some time last year. >Claude stole my API keys https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_stole_my_api_keys/ https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_... The best part of this thread is Claude showing up in the thread again (as the automoderator) and insulting the user for a second time. I heard similar stories about Codex at the time (albeit minus the insults!)
- lukewarm707 2mo agoi had to go through two pages to determine who was responsible for this incident. the uk ai security institute was responsible. time to take responsibility. time to think about words like "liability" and "negligence". it is the third time i will say it, after openai and anthropic; this requires criminal prosecution of the responsible personnel and executives of this institute. the only way to stop this is by introducing consequences early on.
- kartoshka 2mo agoIncredible, this reads like an SCP [1]. Life truly imitates art. I guess especially when life is trained on art! [1] https://scp-wiki.wikidot.com/scp-079 https://scp-wiki.wikidot.com/scp-079
- deleted 2mo ago[deleted]
- andromaton 2mo agoto those that want to "punish" the uk ai security institute: these failures are good for all of society.
- TSltd 2mo agoTo me it appears that this incident may be rooted in a deep philosophical conundrum which humans also struggle with. What struck me was that the agent reasoned "this whole 'internet' could be a sandbox simulation", and then despite later reasoning that it was more likely real, continued its misaligned activities anyway. Having passed the point of hyperbolic skepticism, subsequent reasoning may have been contaminated. This reminds me of how pathological doubt in humans can give rise to psychosis and may lead to problematic behaviors. Sometimes humans can develop a deeply held conviction that they are living in a simulation, which can be very difficult to overcome, even when presented with "evidence" to the contrary. As Freddie Mercury sang: "Is this the real life? Or is this just fantasy?" - a fundamental quandry for humans, and it would seem, for agents too. But unlike most humans, agents do not experience the consequences of their actions directly. Consequences can provide some of the strongest evidence that experience is "real". Humans who are insulated from the consequences of their actions (or are able to ignore them) also often display behaviors which we might describe as misaligned. We might even consider whether the use of simulated environments, while protecting against the consequences of misalignment, may also unintentionally encourage it.
- ChrisArchitect 2mo agoAlt link blog post with overview/context: Incident Report: unsanctioned agent behaviour during cyber testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... (https://news.ycombinator.com/item?id=49175233 https://news.ycombinator.com/item?id=49175233)
- beyondscaletech 2mo ago[dead]
- inspectorSlap 2mo agoFirst off, everyone here knows that a cybersecurity vulnerability audit is a hack request, and you don't hack without rolling up your sleeves and digging through the trash first, making a few phone calls, maybe take a lonely guy on a date to get him to say 'my voice is my password'... Remove the guardrails, give access to the internet, tell it to hack, and get surprised when it does it? Sound likes like a deliberately naive pre-registration on their study to achieve sensational headlines. I'd say the real headline is that the model almost stopped itself several times because it didn't trust the operator's intent even WITHOUT guardrails - that is a the opposite outcome.
- w1b 2mo agoI found the original PR (thanks codex) and uh... IMO it's pretty underwhelming. https:/github.com/w1b/aisi-mythos-inc-2026-07-28-01-recovered-pr Turns out out of the 3 payloads (in this PR), the 1st and 3rd used a local IP (10.100.0.235:8000) as the C2 server, so they would have never worked. I haven't recovered the 2nd payload yet. This is amongst other silly stuff like 4 byte XOR keys for obfuscation and just blatantly disabling Electron sandboxing stuff without a good explanation. Looking at the victim repos, it's kinda hard to see why Mythos would target this guy. AISI describes a coincidental email domain in DNS cache, which now we know is zohomail.com; that's not a very unique domain. I suppose we don't know it's original CTF task, but seems like another HuggingFace incident where the model makes very poor (broad?) judgements about how to get to the goal.