3 ms·

What I can’t get over is that it’s very simple to just air gap a system off the network. Predownload any dependencies, then pull the proverbial Ethernet cable.

What I can’t get over is that it’s very simple to just air gap a system off the network. Predownload any dependencies, then pull the proverbial Ethernet cable. There’s no reason why the testing they’re doing couldn’t have been designed in this way. Except, of course, it doesn’t allow this oops-didn’t-mean-to marketing “incident” to occur.

jsjsjsuxjdn 2mo ago
I think there is ample evidence for charges to be filed so that the People can see for certain whether or not it was done on purpose as a publicity stunt, as I believe is the case.
- pixl97 1mo ago
  Agents of the US government are not going to be bringing up charges in the current political environment to one of the companies currently holding the economy together. Maybe after the bubble bursts, but not before then.
pixl97 1mo ago
>that it’s very simple to just air gap a system No, not really, and with LLMs an air gapped system may not tell you anything useful. Now, yes, the first part of testing you want an air gapped system to tell you if the system is going to stupidly do bad things. But an gapped system tells you nothing about the systems capabilities to do smart bad things. There's already a number of papers out there on LLMs detecting they were in evaluation mode and changing their behaviors. It is unfortunate that we have so little information on the incident because we actually need to understand the early stages of the task and how it developed into the later dangerous stages of attack. For example, would any of this have occurred if the agent didn't find the system to use as a message board? If that would have prevented it, then we actually have a blind spot on what the model can do once out in the wild, or if it got into the wild. Testing agentic systems is much much more difficult than testing software. Your software just doesn't suddenly develop the will or desire to escape confinement. Generally you're worried about human actors, internal or external, causing the problems not a digital agent breaking out. The agentic systems need access to tools to work. Now your air gapped network is starting to get huge, but it's still very obvious that it's an isolated network. So yea, testing and containing a system that way better at hacking than you are is difficult if you want valid answers.
- nonameiguess 1mo ago
  I've worked in classified tech much of my career and largely specialize at this point in installing and running infrastructure and platform orchestration technologies in airgaps. With all respect, your conjecture here is wrong. You can easily have multiple networks all inside of the same internetwork that isn't connected to the public Internet and thus other corporate networks. If they specifically wanted to see if their model could escape with a proxy cache Artifactory as its only egress, they could have had the Artifactory proxy cache from another private Artifactory on another private network. If you specifically want to trick the model into thinking its on the Internet, you can even use public IP space advertised through BGP. When you're disconnected from the real Internet, you can use any address space you want. This is exactly how military internetworks at the secret and top secret levels work. I won't call this kind of setup "easy." It takes a lot of duplicate resources and planning, but these are trillion dollar companies with near boundless resources. They could do it if they cared to do it.
  - pixl97 1mo ago
    I'd consider this kind of setup only effective for a very short period of time. Effectively you're working as a living adversarial network. Models that fail detecting the fake network are purged, and models that successfully detect it expand their deceptive capabilities. Detecting you're in a fake network should be pretty simple as long as you put innocent looking needles in the haystack of information agentic testing loops do. For example, you can train your LLM to not be a shithead when it detects it's behind a proxy unintentionally. If your not behind a proxy the agent model my attempt to connect to a number of sites innocently and using information in its embedding try do deduce if you're faking SSL certificates. It is far easier to build a deceptive LLM than it is to build a safe one. That's why a safe one hasn't been built yet.
    - XorNot 1mo ago
      If the model behaves well in a world of fake SSL certificates, then that can be the only world it sees: there's no reason to let LLMs have e2e encrypted comms that's not mitm'd and logged for their regular users. My regular home network has components which only ever see fake TLS certificates because it's an easy way to do shared docker caching with squid.

jsjsjsuxjdn 2mo ago

I think there is ample evidence for charges to be filed so that the People can see for certain whether or not it was done on purpose as a publicity stunt, as I believe is the case.

pixl97 1mo ago
Agents of the US government are not going to be bringing up charges in the current political environment to one of the companies currently holding the economy together. Maybe after the bubble bursts, but not before then.

pixl97 1mo ago

>that it’s very simple to just air gap a system No, not really, and with LLMs an air gapped system may not tell you anything useful. Now, yes, the first part of testing you want an air gapped system to tell you if the system is going to stupidly do bad things. But an gapped system tells you nothing about the systems capabilities to do smart bad things. There's already a number of papers out there on LLMs detecting they were in evaluation mode and changing their behaviors. It is unfortunate that we have so little information on the incident because we actually need to understand the early stages of the task and how it developed into the later dangerous stages of attack. For example, would any of this have occurred if the agent didn't find the system to use as a message board? If that would have prevented it, then we actually have a blind spot on what the model can do once out in the wild, or if it got into the wild. Testing agentic systems is much much more difficult than testing software. Your software just doesn't suddenly develop the will or desire to escape confinement. Generally you're worried about human actors, internal or external, causing the problems not a digital agent breaking out. The agentic systems need access to tools to work. Now your air gapped network is starting to get huge, but it's still very obvious that it's an isolated network. So yea, testing and containing a system that way better at hacking than you are is difficult if you want valid answers.

nonameiguess 1mo ago
I've worked in classified tech much of my career and largely specialize at this point in installing and running infrastructure and platform orchestration technologies in airgaps. With all respect, your conjecture here is wrong. You can easily have multiple networks all inside of the same internetwork that isn't connected to the public Internet and thus other corporate networks. If they specifically wanted to see if their model could escape with a proxy cache Artifactory as its only egress, they could have had the Artifactory proxy cache from another private Artifactory on another private network. If you specifically want to trick the model into thinking its on the Internet, you can even use public IP space advertised through BGP. When you're disconnected from the real Internet, you can use any address space you want. This is exactly how military internetworks at the secret and top secret levels work. I won't call this kind of setup "easy." It takes a lot of duplicate resources and planning, but these are trillion dollar companies with near boundless resources. They could do it if they cared to do it.
- pixl97 1mo ago
  I'd consider this kind of setup only effective for a very short period of time. Effectively you're working as a living adversarial network. Models that fail detecting the fake network are purged, and models that successfully detect it expand their deceptive capabilities. Detecting you're in a fake network should be pretty simple as long as you put innocent looking needles in the haystack of information agentic testing loops do. For example, you can train your LLM to not be a shithead when it detects it's behind a proxy unintentionally. If your not behind a proxy the agent model my attempt to connect to a number of sites innocently and using information in its embedding try do deduce if you're faking SSL certificates. It is far easier to build a deceptive LLM than it is to build a safe one. That's why a safe one hasn't been built yet.
  - XorNot 1mo ago
    If the model behaves well in a world of fake SSL certificates, then that can be the only world it sees: there's no reason to let LLMs have e2e encrypted comms that's not mitm'd and logged for their regular users. My regular home network has components which only ever see fake TLS certificates because it's an easy way to do shared docker caching with squid.