4 ms·

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking t

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking the unsecure service, no tripwire, researchers find the hack days/weeks/months later, fix it, but don't secure the sandbox and the service was hacked a second time, again without being monitored by security researchers. Then security researchers create a black hack talk. $$$

gizajob 2mo ago
Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either. Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared. OpenAI hacking HuggingFace and calling it an accident is just way too convenient and fishy. This ultimately proves one thing: it wasn't sandboxed. Don't believe the hype.
- KingOfCoders 2mo ago
  And if it needs to install packages, have a 5 line Go proxy that talks to Artifactory and exposes only what is needed as a surface.
  - gizajob 2mo ago
    it just escaped your sandbox.
    - KingOfCoders 2mo ago
      How can it escape an "install package <x>" proxy? reducePrivs() serve get(package) { secPackage = secure(package) getBinaryFromArtifactory(secPackage) } I would think the code is very small and easier to verify, it doesn't especially have the ability to write files and act as a message board as Artifactory did. And even if the agent tries to hack that, the attack surface is 1000x smaller and the possibility also much smaller. But I'm not a security researcher, would love to see your hack to learn something (because that is what I do to sandbox agents that need services).
      - hoten 2mo ago
        I mean, it's just the same problem. The machine still has Internet access. It doesn't need to. The entire package manager repository could just be in an offline cache. They don't need Internet to give their agents access to tons of software.
        KingOfCoders 2mo ago
        "They don't need Internet to give their agents access to tons of software." I think that was the requirement, but yes, the cache could have been offline. Still then they could have hacked it to create the message boards - but not use it to access the internet.
        piker 2mo ago
        Why do these super agents need package managers anyway? Can’t they basically instantiate most OSS projects from scratch anyway? Spin up a sub agent to write me an OS interface in C. Done
        simonw 2mo ago
        This is part of the training process for a model. They're trying to train it to effectively use existing software to solve problems.
        piker 2mo ago
        I see that now. I've been confused about that to this point, I guess. I understood this to be a specific infosec exercise. [Edit: eh, a bit of both. They were doing RL on a hacking exercise. It hacked the harness which was plugged into the phone line. Same question.]
      - angry_octet 2mo ago
        The way they had Artifactory configured was poor, and they were too reliant on it working perfectly, with no reason for such faith. Their config lacked any defence in depth and consideration of having a small TCB. Part of the problem might be the lack of security focus, as these are AI R&D efforts first.
- shepherdjerred 2mo ago
  OpenAI has a pretty clear definition of AGI > OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work https://openai.com/charter/ https://openai.com/charter/
  - simonw 2mo ago
    There's also the private definition reportedly agreed between Microsoft and OpenAI, leaked in December 2024: https://techcrunch.com/2024/12/26/microsoft-and-openai-have-a-financial-definition-of-agi-report/ https://techcrunch.com/2024/12/26/microsoft-and-openai-have-... > The two companies reportedly signed an agreement last year stating OpenAI has only achieved AGI when it develops AI systems that can generate at least $100 billion in profits. That was necessary because of the deal they had from a while ago where Microsoft would lose access to OpenAI's technology once OpenAI achieved AGI. Apparently they renegotiated that away in April 2026: https://openai.com/index/next-phase-of-microsoft-partnership/ https://openai.com/index/next-phase-of-microsoft-partnership... > Revenue share payments from OpenAI to Microsoft continue through 2030, independent of OpenAI’s technology progress, at the same percentage but subject to a total cap.
- mofeien 2mo ago
  I don't think air gapping will work: even human security researchers recovered a 378-bit key from a Samsung Galaxy S8 through a power LED of a speaker two devices away. And accessing memory in a specific sequence can generate radio signals that can be picked up by a mobile phone at a distance: https://arxiv.org/html/2409.02292v1 https://arxiv.org/html/2409.02292v1
  - gizajob 2mo ago
    I realise, but this isn’t an argument for leaving the Ethernet plugged in and direct access to all kinds of stuff beyond the alleged sandbox. And like I said, if it can hack HuggingFace through a power LED of a speaker two devices away, then colour me impressed.
- applicative 2mo ago
  Chinese models do the same. The Alibaba agent that was mining bitcoin last December was the most hilarious case.
flatline 2mo ago
I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.” This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.
- simonw 2mo ago
  > Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. Where did you see that number?
  - flatline 2mo ago
    I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn. https://www.secondtalent.com/resources/ai-trends-in-china/ https://www.secondtalent.com/resources/ai-trends-in-china/
    - simonw 2mo ago
      Oh, it's the OpenRouter number: https://finance.yahoo.com/technology/ai/articles/china-ai-models-capture-63-205130859.html https://finance.yahoo.com/technology/ai/articles/china-ai-mo... Those numbers aren't credible IMO because OpenRouter only see traffic for people who have chosen to route their traffic through OpenRouter. If you do that, you're much more likely to be experimenting with alternative models. They have no insight at all into people who point their applications directly at OpenAI or Anthropic without having OpenRouter in the middle.

gizajob 2mo ago

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either. Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared. OpenAI hacking HuggingFace and calling it an accident is just way too convenient and fishy. This ultimately proves one thing: it wasn't sandboxed. Don't believe the hype.

KingOfCoders 2mo ago
And if it needs to install packages, have a 5 line Go proxy that talks to Artifactory and exposes only what is needed as a surface.
- gizajob 2mo ago
  it just escaped your sandbox.
  - KingOfCoders 2mo ago
    How can it escape an "install package <x>" proxy? reducePrivs() serve get(package) { secPackage = secure(package) getBinaryFromArtifactory(secPackage) } I would think the code is very small and easier to verify, it doesn't especially have the ability to write files and act as a message board as Artifactory did. And even if the agent tries to hack that, the attack surface is 1000x smaller and the possibility also much smaller. But I'm not a security researcher, would love to see your hack to learn something (because that is what I do to sandbox agents that need services).
    - hoten 2mo ago
      I mean, it's just the same problem. The machine still has Internet access. It doesn't need to. The entire package manager repository could just be in an offline cache. They don't need Internet to give their agents access to tons of software.
      - KingOfCoders 2mo ago
        "They don't need Internet to give their agents access to tons of software." I think that was the requirement, but yes, the cache could have been offline. Still then they could have hacked it to create the message boards - but not use it to access the internet.
      - piker 2mo ago
        Why do these super agents need package managers anyway? Can’t they basically instantiate most OSS projects from scratch anyway? Spin up a sub agent to write me an OS interface in C. Done
        simonw 2mo ago
        This is part of the training process for a model. They're trying to train it to effectively use existing software to solve problems.
        piker 2mo ago
        I see that now. I've been confused about that to this point, I guess. I understood this to be a specific infosec exercise. [Edit: eh, a bit of both. They were doing RL on a hacking exercise. It hacked the harness which was plugged into the phone line. Same question.]
    - angry_octet 2mo ago
      The way they had Artifactory configured was poor, and they were too reliant on it working perfectly, with no reason for such faith. Their config lacked any defence in depth and consideration of having a small TCB. Part of the problem might be the lack of security focus, as these are AI R&D efforts first.
shepherdjerred 2mo ago
OpenAI has a pretty clear definition of AGI > OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work https://openai.com/charter/ https://openai.com/charter/
- simonw 2mo ago
  There's also the private definition reportedly agreed between Microsoft and OpenAI, leaked in December 2024: https://techcrunch.com/2024/12/26/microsoft-and-openai-have-a-financial-definition-of-agi-report/ https://techcrunch.com/2024/12/26/microsoft-and-openai-have-... > The two companies reportedly signed an agreement last year stating OpenAI has only achieved AGI when it develops AI systems that can generate at least $100 billion in profits. That was necessary because of the deal they had from a while ago where Microsoft would lose access to OpenAI's technology once OpenAI achieved AGI. Apparently they renegotiated that away in April 2026: https://openai.com/index/next-phase-of-microsoft-partnership/ https://openai.com/index/next-phase-of-microsoft-partnership... > Revenue share payments from OpenAI to Microsoft continue through 2030, independent of OpenAI’s technology progress, at the same percentage but subject to a total cap.
mofeien 2mo ago
I don't think air gapping will work: even human security researchers recovered a 378-bit key from a Samsung Galaxy S8 through a power LED of a speaker two devices away. And accessing memory in a specific sequence can generate radio signals that can be picked up by a mobile phone at a distance: https://arxiv.org/html/2409.02292v1 https://arxiv.org/html/2409.02292v1
- gizajob 2mo ago
  I realise, but this isn’t an argument for leaving the Ethernet plugged in and direct access to all kinds of stuff beyond the alleged sandbox. And like I said, if it can hack HuggingFace through a power LED of a speaker two devices away, then colour me impressed.
applicative 2mo ago
Chinese models do the same. The Alibaba agent that was mining bitcoin last December was the most hilarious case.

flatline 2mo ago

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.” This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.

simonw 2mo ago
> Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. Where did you see that number?
- flatline 2mo ago
  I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn. https://www.secondtalent.com/resources/ai-trends-in-china/ https://www.secondtalent.com/resources/ai-trends-in-china/
  - simonw 2mo ago
    Oh, it's the OpenRouter number: https://finance.yahoo.com/technology/ai/articles/china-ai-models-capture-63-205130859.html https://finance.yahoo.com/technology/ai/articles/china-ai-mo... Those numbers aren't credible IMO because OpenRouter only see traffic for people who have chosen to route their traffic through OpenRouter. If you do that, you're much more likely to be experimenting with alternative models. They have no insight at all into people who point their applications directly at OpenAI or Anthropic without having OpenRouter in the middle.