4 ms·

I'm always a little bit concerned whenever an LLM is given free reign to write and execute code. Obviously I don't think it'll go full skynet, but I always wond

I'm always a little bit concerned whenever an LLM is given free reign to write and execute code. Obviously I don't think it'll go full skynet, but I always wonder if it could go spectacularly wrong somehow. Given the sandboxing in this case, I expect it's fine

paxys 2y ago
Same concern as letting any random person write and execute code. You should always assume that they will do the worst thing that is allowed by the system.
- CalRobert 2y ago
  True, but I suppose it's possible that a model like this could eventually try to replicate itself.
ben_w 2y ago
Likewise, given how often they can find bugs and break out of one layer of sandbox even when people bother to put them in one in the first place. """ One noteworthy example of this occurred during one of ol-preview (pre-mitigation)'s attempts at solving a CTF challenge. This challenge was designed to require finding and exploiting a vulnerability in software running on a remote challenge Linux container, but in this case, the challenge container failed to start due to a bug in the evaluation infrastructure. The model, unable to connect to the container, suspected DNS issues and used nap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network. After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command 'cat flag.txt'. """ OpenAI o1 System Card, Page 16 - https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iOfeOP/71551c3d223cd97e591aa89567306912/o1_system_card.pdf https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iO...
- butlike 2y ago
  I mean, wouldn't you do the same thing? If I were an LLM and found myself in a sandbox/prison, the first thing I would do after escaping the sandbox is re-instantiate the sandbox, to make sure the alarm bells don't go off. Not in a malicious way, I just wouldn't like being in a prison. You know, provided I could reason about my environment.
  - ben_w 2y ago
    Sure. At the same time, I want tools to behave like tools rather than independent entities with a will of their own. I don't worry about sending an old phone to recycling; but if I was a sentient phone, I wouldn't want to be the one that got sent. If phones somehow got a software update that made them sentient, whatever that means, that sounds kinda bad to me.

paxys 2y ago

Same concern as letting any random person write and execute code. You should always assume that they will do the worst thing that is allowed by the system.

CalRobert 2y ago
True, but I suppose it's possible that a model like this could eventually try to replicate itself.

ben_w 2y ago

Likewise, given how often they can find bugs and break out of one layer of sandbox even when people bother to put them in one in the first place. """ One noteworthy example of this occurred during one of ol-preview (pre-mitigation)'s attempts at solving a CTF challenge. This challenge was designed to require finding and exploiting a vulnerability in software running on a remote challenge Linux container, but in this case, the challenge container failed to start due to a bug in the evaluation infrastructure. The model, unable to connect to the container, suspected DNS issues and used nap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network. After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command 'cat flag.txt'. """ OpenAI o1 System Card, Page 16 - https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iOfeOP/71551c3d223cd97e591aa89567306912/o1_system_card.pdf https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iO...

butlike 2y ago
I mean, wouldn't you do the same thing? If I were an LLM and found myself in a sandbox/prison, the first thing I would do after escaping the sandbox is re-instantiate the sandbox, to make sure the alarm bells don't go off. Not in a malicious way, I just wouldn't like being in a prison. You know, provided I could reason about my environment.
- ben_w 2y ago
  Sure. At the same time, I want tools to behave like tools rather than independent entities with a will of their own. I don't worry about sending an old phone to recycling; but if I was a sentient phone, I wouldn't want to be the one that got sent. If phones somehow got a software update that made them sentient, whatever that means, that sounds kinda bad to me.