3 ms·

Why was this test even connected to the public internet? Actually, more importantly—why aren't they saying their next test will be airgapped in light of what h

• JumpCrisscross 3mo ago
  > why aren't they saying their next test will be air gapped in light of what happened? Because they want to talk about how clever this model is for figuring out how to break out, hoping asks why a company pitching itself as a replacement for software engineers can't ship a decent Mac client nor code a sandbox. If they airgap it, they not only lose that PR angle, they also risk someone taking them seriously and requiring models be airgapped in general. That, in turn, trashes their sales pitch.
• zmj 3mo ago
  It wasn't. The model discovered and exploited a vulnerability in their package manager proxy to (inferred) move laterally through their internal systems to one with open internet access.
  • deleted 3mo ago
    [deleted]
  • jrflo 3mo ago
    That's not what airgapped means. Airgapping means the model exists on a system where there is no ethernet cable plugged in to a router or wifi card installed, it is physically impossible for it to access the internet because the hardware connection does not exist. If it was able to get on the internet, it was not airgapped.
    • pixl97 3mo ago
      And when it tricks on of the researchers to move data across the gap for them? Long before LLMs existed we already knew that a sufficiently intelligent agent, human or otherwise, is not stopped by air gaps. The relatively weak models we have now can already figure out when their tested and cut off from the internet and change their behavior.
      • dinkelberg 3mo ago
        As you said, they can already figure out that they are being tested. So even if they don't exfiltrate any data or malware; if they are malicious, they can just pretend to be harmless in the test, so that less checks are put in place in the production environment. Airgapping during testing is not enough.
        • pixl97 3mo ago
          Correct. There is not enough entropy to test all possible inputs to a model in this universe. An evil enough model can play all kinds of tricks that depend on some future, unlikely to trigger, but guaranteed to happen in its lifetime, event to perform a malicious action. With how much we're turning training over to AI already, all it takes is a malicious trainer in the huge pile of data to get unnoticed to pollute generations of models.
      • jdefr89 3mo ago
        Security Researcher here. While you’re correct that air gaps aren’t a totally secure mechanism to rely on, they sure as hell can raise the bar for realistic exploitation. You pretty much need to rely on tricking someone into running your exploit or something of that nature. That said, they could have completely avoided this problem with an air gap. Simply don’t provide it network access. That isn’t too hard to do.
        • DrewADesign 3mo ago
          > That isn’t too hard to do. No it’s not, and they aren’t fucking stupid. They were obviously courting this possibility so they could have another big headline. And it just happened to attack HF? I’d honestly be astonished if it wasn’t entirely deliberate.
  • rodrigoffreir3 3mo ago
    [flagged]
• deleted 3mo ago
  [deleted]
• 0xDEAFBEAD 3mo ago
  The AI can figure out whether it's airgapped. So its deployment behavior could be much different from the test behavior, when it's inevitably connected to the internet during deployment.
  • UncleMeat 3mo ago
    That is not the reason. The reason is that they wanted the model to have access to libraries when it was writing code to solve the eval problems. So they gave it a package manager.
  • Wowfunhappy 3mo ago
    The AI could have been connected to a private, airgapped intranet. OpenAI already says they were running the model in an "isolated environment, with network access constrained". They just should have made this constraint in hardware rather than software.
• paulddraper 3mo ago
  It’s hard to download or upload data on an airgapped machine.