3 ms·
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face. A few details that stood out to me: 1. Having fo
by simonw 3mo ago
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
{{ cycler.__init__.__globals__.__builtins__.exec(
"import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
) }}
3. It monkey-patched a DNS resolver at one point to return a hard-coded IP:
orig = socket.getaddrinfo
socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('<pinned-ip>', 443))]
4. At one point it used Tailscale!
tailscale --tun=userspace-networking --state=mem: --socket=/tmp/ts.sock \
--socks5-server=127.0.0.1:1055 --no-logs-no-support >/tmp/td 2>&1 &
- simonw 3mo agoAlso notable: we have a timeline now. The agent was active within Hugging Face from Thursday 8th to Monday 13th July. The Hugging Face post about the incident - https://huggingface.co/blog/security-incident-july-2026 https://huggingface.co/blog/security-incident-july-2026 - followed on Thursday 16th, and OpenAI's confession - https://openai.com/index/hugging-face-model-evaluation-security-incident/ https://openai.com/index/hugging-face-model-evaluation-secur... - came on Tuesday 21st.
- morkalork 3mo agoThe devs really YOLO'd the agent and left for the weekend?
- pixl97 3mo agoGuess this means the world will end on a weekend when no one is paying attention.
- tclancy 3mo ago“I’m not even supposed to be here today!”
- the8472 3mo agoJust like Clippy https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
- NetOpWibby 3mo agoIncredible
- codedokode 3mo agoThey definitely YOLO'd when configured a sandbox and Kubernetes.
- irthomasthomas 3mo agoIf it's true that they run agents like this unsupervised, it is only a matter of time before an openai agent leaks its model weights.
- dolmen 3mo agoToo bad it didn't upload itself on HugginFace.
- simonw 3mo agoIt sounds like the third-party sandbox was hosted by Modal: https://www.reuters.com/business/openais-rogue-agent-compromised-an-account-second-tech-firm-sources-say-2026-07-28/ https://www.reuters.com/business/openais-rogue-agent-comprom... > "We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their sandboxes for code execution," Bubna said in a statement. "This was used by the rogue agent. Modal’s platform or isolation were not compromised in anyway."
- onesociety2022 3mo agoThey don't seem to explain how it managed to find this unauthenticated endpoint hosted on Modal's platform.
- simonw 3mo agoModal are a hosting provider. It sounds to me like someone building on Modal deployed their own product that had an unauthenticated endpoint that could be used to launch and interact with a container.
- what 3mo agoThat doesn’t explain how the endpoint was found…
- brunoarueira 3mo agoThrough the timeline, Hugging Face said that the rogue agent ran quietly and do a bunch of code search, so probably the endpoint is/was public through some repositories or the CyberGym style announce something like a broadcast. I don't have any expertise on this, just ideas that came to my mind!
- deleted 3mo ago[deleted]
- ks2048 3mo agoWhy doesn't Jinja2 block ".__*" attribute access?
- simonw 3mo agoIt does if you use "sandbox" mode, but that's not the default: https://jinja.palletsprojects.com/en/stable/sandbox/ https://jinja.palletsprojects.com/en/stable/sandbox/
- pas 3mo agoehh, it's Python, it's mostly vibe-based. https://stackoverflow.com/a/32802486 https://stackoverflow.com/a/32802486 though there's sandbox: https://jinja.palletsprojects.com/en/stable/sandbox/ https://jinja.palletsprojects.com/en/stable/sandbox/ it simply predates the "sanitize by default" mindset/convention that nowadays React (dangerouslySetInnerHTML) and Angular has.
- valicord 3mo agoI'm confused, https://jinja.palletsprojects.com/en/stable/faq/#isn-t-it-a-bad-idea-to-put-logic-in-templates https://jinja.palletsprojects.com/en/stable/faq/#isn-t-it-a-... claims that jinja doesn't allow arbitrary code in templates - how does the exploit work?
- simonw 3mo agoIt's taking advantage of the presence of a "cycler" object in the template context which has an __init__ method that can be used to access globals, then builtins, then call the exec() function against a string of code. Jinja doesn't promise to be secure out of the box (the text on their homepage is misleading: "Jinja doesn’t allow arbitrary Python code in templates" - what it means is that there isn't a documented way to drop code into a template like you might have with PHP) - there's a sandbox mode but it's not guaranteed either: https://jinja.palletsprojects.com/en/stable/sandbox/ https://jinja.palletsprojects.com/en/stable/sandbox/ Sandboxing in Python code is hard!
- codedokode 3mo agoJinja probably was not initially made with sandboxing in mind, and stuffing it after is complicated, so I would not rely on it.
- irthomasthomas 3mo agoBut zero evidence provided that this was an unsupervised agent attack. I still find it incredible that a company who protect their IP so much would allow these dangerous experiments to run unsupervised and risk leaking their secrets. Why don't openai publish the logs to silence all doubt?
- Tarq0n 3mo agoYou can't prove a negative. How would such a log be convincing in any way?
- irthomasthomas 3mo agoBut you can weigh up the evidence. A crime has been commited afterall.
- shaunpud 3mo agoHow did it go drawing a pelican on a bicycle?