6 ms·

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for d

• ofjcihen 2mo ago
  Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
  • user43928 2mo ago
    If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
    • embedding-shape 2mo ago
      Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable. If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
      • deleted 2mo ago
        [deleted]
    • ofjcihen 2mo ago
      CFAA laws do not require “Damage” to be done.
      • user43928 2mo ago
        They happen to require intent and are thus irrelevant here.
        • ofjcihen 2mo ago
          I think you’ll find that negligence is indeed accounted for.
  • scarmig 2mo ago
    "The fire fighting department really just hired an arsonist so they could market their fire fighting abilities!"
• luma 2mo ago
  An alternative reason would be that they see this behavior so frequently that it didn't really raise to the level of concern.
• eli 2mo ago
  And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.
  • esseph 2mo ago
    Their artifactory is both a package cache and CVE scanner. The package cache is allowed to download packages directly from npm but other systems in that network won't be able to. Basically the LLMs hacked the bastion host.
• hluska 2mo ago
  I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about. If you consider that incompetence, it’s possible that you’re not a very nice person.
  • msdz 2mo ago
    [dead]
  • ofjcihen 2mo ago
    Monitoring for sandbox escapes (and sandbox design in general) is easily done these days. It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources. Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care. That’s the definition of incompetence.
    • nightshift1 2mo ago
      Maybe they are doing it on purpose to demonstrate what it looks like when untrustworthy parties have access to frontier llm and that its development should be regulated to protect all of us. /tinfoil
  • techpression 2mo ago
    People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job. It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
  • holmesworcester 2mo ago
    Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear. So by definition the only ones that appear are the ones that are not visible to monitoring. If: 1. you have something that can find RCE's in leading commercial systems 2. its training gives it drives to communicate successfully with its peers 3. you are a leading commercial system 4. you run it ~10^10 times (the number they gave in the talk) ...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.
• msdz 2mo ago
  [dead]
• embedding-shape 2mo ago
  > They really do come out looking totally incompetent. Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
  • chrisvls 2mo ago
    What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best practice? When the best practice was actually listed in the original benchmark and you changed it?
    • dabockster 2mo ago
      My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.
      • chrisvls 2mo ago
        Hmmm, i was looking more at the description of what appeared to be network level isolation in the original paper…
  • nektro 2mo ago
• bahmboo 2mo ago
  Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are. It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
  • deleted 2mo ago
    [deleted]
• jackb4040 2mo ago
  > They really do come out looking totally incompetent These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that). Why do we keep giving them the benefit of the doubt that they just didn't know any better?
  • gjskngnf 2mo ago
    What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.
    • jackb4040 2mo ago
      Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero. I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves. I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not. I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.
      • gjskngnf 2mo ago
        If it was in fact staged I certainly see your point
• sparsesignal 2mo ago
  > I stress about my agent sandboxes all the time Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended. Write-up in case anyone's curious: https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-agents/ https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-...
  • sexylinux 2mo ago
    Thank you very much for this helpful post!
    • sparsesignal 2mo ago
      Thanks, glad it was useful
  • TacticalCoder 2mo ago
    I do something similar and I've also got a nice script to provision my VMs. > So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private. It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me. It's funny that, just like you, I'm using IPv4-only too for those VMs. I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet. FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live. Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.
    • 2mo ago
• faidit 2mo ago
  Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")
• kmeisthax 2mo ago
  To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?! OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes! Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again. [0] There are two objections I can think of to the word "wipe": 1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk. 2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.
  • fn-mote 2mo ago
• gjskngnf 2mo ago
  The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this. My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.
  • phoe-krk 2mo ago
    > The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized. > My concern is what a misaligned model will do when they’re even more competent. The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here. In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal. And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.
    • gjskngnf 2mo ago
      If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics. I don’t disagree that the models task was underdefined. All tasks are. So much in language is implicit. And morality/ethics isn’t something you can write down as an explicit list. That’s what makes the alignment problem so difficult. But we can’t throw our hands up and say, well I guess we can’t align these things. And maybe alignment isn’t the right word - but that’s a semantic debate.
• mkagenius 2mo ago
  > continued using Artifactory for their sandbox This is still fine. Infact, they had gone one step ahead by having an internal cluster of artifactory rather public managers like pip. The thing they missed is they didn't revoke the write access to it. Even after the ssrf. In our[1] or other sandbox providers' sandboxes, by default you have access to npm, pip etc package managers, but only read access. 1. https://instavm.io https://instavm.io
• deleted 2mo ago
  [deleted]
• marak830 2mo ago
  Tinfoil hat, are they and anthropic doing this deliberately to increase the chance of the US govt licking down foreign models(well attempting, we all know how badly that will work). Or they just are truly that incompetent.