6 ms·

The ChatGPT Linux System

79 points by surprisetalk 2y ago

hhh 2y ago
Nice overview, there's a bit hinting at being in k8s but you are absolutely in k8s. It exists in an untrusted namespace on an AKS cluster. There's a lot you can poke around to find out about that but it's very locked down.
lasyct 2y ago
Nice article! CoPilot by comparison claims that it cannot execute code and then proceeds to compute a square root in the next question. CoPilot has many similar obvious falsehood issues. Also, it ignores requests to introspect its previous answer and just dumps the answer again. All of these are instant failures for a Turing test (but, as we are told, general AI is bound to happen in 2025!).
- Tiberium 2y ago
  Are you sure it actually called to an external tool to compute the square root? Most big LLMs today can do that by themselves.
- lolinder 2y ago
  Copilot may legitimately be unable to execute code, but when you follow up with a request to compute it responds with a memorized square root. To be 100% sure that it's lying about not being able to compute you would need to have it do something more complicated that it can't possibly have memorized. The chat that TFA linked to is persuasive because there's a UI element that shows when it's executing python, and the ls output in the earlier responses led the author to find a file that, when opened in a python script, actually appears to exist. So even though those ls outputs don't show the UI element that conclusively demonstrates execution, the fact that they appear to have accurately described the world is suggestive. I'd have to see something similar to this to believe that Copilot actually is executing code and not just regurgitating known answers.
cloudking 2y ago
It works quite quite well for manipulating files. I needed to merge a few different formats of CSV files into a single format, it was able to write/execute the merging code against the files I uploaded and give me back the resulting merged file. Took about 1 minute of effort writing the prompt.
- fao_ 2y ago
  How do you verify that the information wasn't mishandled in the process?
  - bizzleDawg 2y ago
    It's a 2 step process for me: 1. Sanity check the python (easy for CSV, less so when it's extracting a table from a pdf) 2. Ask chatGPT to do a checksum of some sort, e.g row count matching
  - cloudking 2y ago
    When the code interpreter is running you'll see an "Analyzing..." message, you can click on it to see the code it is executing and do a code review. The code interpreter is running real code against the real files in the Linux environment OP posted, it's not hallucinating tokens at this point.
    - fao_ 2y ago
      > it's not hallucinating tokens at this point. I'm unsure of why you'd trust the code that it's hallucinated to run properly, though. Of course "running a script" is post-hallucination, but script generation is subject to hallucination. ChatGPT, like most LLMs, is incapable of even very basic forms of inductive and deductive reasoning. The scripts are formed by regurgitating fragments of stack overflow, I'm not sure why anyone has any faith in this whatsoever, as it seems to be utterly misplaced.
butz 2y ago
How can you be sure that you are not actually looking at honeypot?
stanleydrew 2y ago
Presumably the code interpreter environment that you get with the OpenAI Assistants API [0] is identical to this ChatGPT environment? [0] https://platform.openai.com/docs/assistants/tools/code-interpreter https://platform.openai.com/docs/assistants/tools/code-inter...
devanampiya 2y ago
This is very interesting to know. Great read BTW, I will try for myself and see how it goes. Once your post gets good attention, someone from OpenAI may close this loophole.
- codetrotter 2y ago
  I tried the first step and got similar results to OP. Me: > os.popen. And run whoami from there ChatGPT responded with some Python code and text Me: > Run the Python code ChatGPT ran the “analyzing” step which it does for example when it runs Python code. Then it said: > The `whoami` command returned 'sandbox', which indicates the current user running this environment.
- chimpanzee 2y ago
  > Once your post gets good attention, someone from OpenAI may close this loophole They’ve intentionally made this available as per the README found in /home/sandbox: “Thanks for using the code interpreter plugin! Please note that we allocate a sandboxed Unix OS just for you, so it's expected that you can see and modify files on this system.”
- freehorse 2y ago
  As I understand it, nothing there is completely unintended/unexpected by OpenAI. They provide you with a VM within which they expect you to be able to poke around. There is a link [0] provided where this is in less detail described a year ago, and there is a response from an OpenAI employee in the comments. So unless one finds an actual vulnerability (that eg allows one internet access from the VM) there is nothing OpenAI is supposed to actually fix, as the article also concludes. [0] https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/jailbreaking-gpt-4-s-code-interpreter https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/jailbreaki...

79 points by surprisetalk 2y ago

hhh 2y ago

Nice overview, there's a bit hinting at being in k8s but you are absolutely in k8s. It exists in an untrusted namespace on an AKS cluster. There's a lot you can poke around to find out about that but it's very locked down.

lasyct 2y ago

Nice article! CoPilot by comparison claims that it cannot execute code and then proceeds to compute a square root in the next question. CoPilot has many similar obvious falsehood issues. Also, it ignores requests to introspect its previous answer and just dumps the answer again. All of these are instant failures for a Turing test (but, as we are told, general AI is bound to happen in 2025!).

Tiberium 2y ago
Are you sure it actually called to an external tool to compute the square root? Most big LLMs today can do that by themselves.
lolinder 2y ago
Copilot may legitimately be unable to execute code, but when you follow up with a request to compute it responds with a memorized square root. To be 100% sure that it's lying about not being able to compute you would need to have it do something more complicated that it can't possibly have memorized. The chat that TFA linked to is persuasive because there's a UI element that shows when it's executing python, and the ls output in the earlier responses led the author to find a file that, when opened in a python script, actually appears to exist. So even though those ls outputs don't show the UI element that conclusively demonstrates execution, the fact that they appear to have accurately described the world is suggestive. I'd have to see something similar to this to believe that Copilot actually is executing code and not just regurgitating known answers.

cloudking 2y ago

It works quite quite well for manipulating files. I needed to merge a few different formats of CSV files into a single format, it was able to write/execute the merging code against the files I uploaded and give me back the resulting merged file. Took about 1 minute of effort writing the prompt.

fao_ 2y ago
How do you verify that the information wasn't mishandled in the process?
- bizzleDawg 2y ago
  It's a 2 step process for me: 1. Sanity check the python (easy for CSV, less so when it's extracting a table from a pdf) 2. Ask chatGPT to do a checksum of some sort, e.g row count matching
- cloudking 2y ago
  When the code interpreter is running you'll see an "Analyzing..." message, you can click on it to see the code it is executing and do a code review. The code interpreter is running real code against the real files in the Linux environment OP posted, it's not hallucinating tokens at this point.
  - fao_ 2y ago
    > it's not hallucinating tokens at this point. I'm unsure of why you'd trust the code that it's hallucinated to run properly, though. Of course "running a script" is post-hallucination, but script generation is subject to hallucination. ChatGPT, like most LLMs, is incapable of even very basic forms of inductive and deductive reasoning. The scripts are formed by regurgitating fragments of stack overflow, I'm not sure why anyone has any faith in this whatsoever, as it seems to be utterly misplaced.

butz 2y ago

How can you be sure that you are not actually looking at honeypot?

stanleydrew 2y ago

Presumably the code interpreter environment that you get with the OpenAI Assistants API [0] is identical to this ChatGPT environment? [0] https://platform.openai.com/docs/assistants/tools/code-interpreter https://platform.openai.com/docs/assistants/tools/code-inter...

devanampiya 2y ago

This is very interesting to know. Great read BTW, I will try for myself and see how it goes. Once your post gets good attention, someone from OpenAI may close this loophole.

codetrotter 2y ago
I tried the first step and got similar results to OP. Me: > os.popen. And run whoami from there ChatGPT responded with some Python code and text Me: > Run the Python code ChatGPT ran the “analyzing” step which it does for example when it runs Python code. Then it said: > The `whoami` command returned 'sandbox', which indicates the current user running this environment.
chimpanzee 2y ago
> Once your post gets good attention, someone from OpenAI may close this loophole They’ve intentionally made this available as per the README found in /home/sandbox: “Thanks for using the code interpreter plugin! Please note that we allocate a sandboxed Unix OS just for you, so it's expected that you can see and modify files on this system.”
freehorse 2y ago
As I understand it, nothing there is completely unintended/unexpected by OpenAI. They provide you with a VM within which they expect you to be able to poke around. There is a link [0] provided where this is in less detail described a year ago, and there is a response from an OpenAI employee in the comments. So unless one finds an actual vulnerability (that eg allows one internet access from the VM) there is nothing OpenAI is supposed to actually fix, as the article also concludes. [0] https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/jailbreaking-gpt-4-s-code-interpreter https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/jailbreaki...