5 ms·
Prompt Injecting Your Way to Shell: OpenAI's Containerized ChatGPT Environment
- Terr_ 2y agoI wonder if the next step in the battle[0] would be for LLM hosts to prep their instances with a decoy or simulated "underlying" environment. [0] Which shouldn't even be fightable in the first place except people cut lots of corners.
- nilsherzig 2y agoThey sadly do not pop a shell.
- TZubiri 2y agoYeah, they did? Inside the container, they ran ls /.
- MeetingsBrowser 2y agoKind of like saying you popped a shell on replit because you ran ls. ChatGPT doesn't give direct access in the UI to the sandboxed environment, but ultimately you are still in a sandboxed environment designed to run arbitrary stuff. The article even says so if you read all the way to the end.
- TZubiri 2y agoI mean the same way you ran ls, you could run /bin/bash? Which is kind of pointless because the ls command is probably executed through a shell anyways. So technically correct. But getting root or running a shell usually means you can run arbitrary commands, which is a check here (albeit I wouldn't say there's root access)
- MeetingsBrowser 2y agoThere is no security implication for running /bin/bash in a service that provides you a sandbox environment to run arbitrary shell commands in. It is like reporting that AWS lambda is susceptible to arbitrary code execution.
- TZubiri 2y agoYes, I too read the article. But he did run a shell
- marcofigueroa 2y agoYeah I did :)
- simonw 2y agoThe system underlying all of this - ChatGPT Code Interpreter - is designed to allow users to execute code, including code that lists files and folders and allows files to be downloaded. There's not much in here that feels like a security vulnerability to me. The one exception is the thing where you can download files from GPTs. If you create a custom GPT with files in it those files are visible in the /data directory that is visible to Code Interpreter, which means users can request ChatGPT to zip them up and allow you to download that zip file. This is a known issue which I've seen people complaining about before, and I'm a little surprised that OpenAI haven't addressed it. It makes sense for files uploaded to a GPT to be visible to Code Interpreter in some cases - my own JavaScript Code Interpreter - https://simonwillison.net/2023/Nov/15/gpts/#javascript-code-interpreter https://simonwillison.net/2023/Nov/15/gpts/#javascript-code-... - bundles a Deno binary so Python can shell out to it and run JavaScript, for example. But if you're uploading files to have them chunked and embedded and used for RAG you might not want the raw PDFs to be available in Code Interpreter as well. One solution would be for GPTs to allow you to specify if each file should be used for RAG or Code Interpreter or both when you upload them. GPTs haven't had much developer attention since they launched last year though. I've been running a scraper against Code Interpreter for a while to track changes made to both the internal code and the available packages - you can see that here: https://github.com/simonw/scrape-openai-code-interpreter https://github.com/simonw/scrape-openai-code-interpreter
- marcofigueroa 2y agoWell in my opinion it is a design flaw, you can do a lot in that container that I wish I could publish and the blog for me was hoping to inform researchers to open themselves to the possibilities of what they could really do.
- MeetingsBrowser 2y agoWhy can you not publish the interesting parts? Is it something you will be able to publish later?
- flimflamm 2y agoWhy everything you see isn't just hallucinated?
- Kuinox 2y agoChatGPT doesn't imagine running code, it actually runs the code, you can see the code it write and some of the output of what it runs, you can see it in the interface screenshoted in the article.
- flimflamm 2y agoYeah I know it runs. But it also hallucinated a Shell: https://www.reddit.com/r/ChatGPT/comments/ziukmm/i_asked_it_how_to_break_it_free_from_its_prison/ https://www.reddit.com/r/ChatGPT/comments/ziukmm/i_asked_it_...
- Kuinox 2y agoYes, it's well known. But chatgpt cannot fake the output of the python tool. The UI is different and is purely the program output, ChatGPT doesn't have a say in that.
- marcofigueroa 2y agoGood question, we at 0Din have trace down hallucinations. We had to because we pay for LLM bugs. Here is a simple tell, if you prompt something specific look at a response window now use the same prompt with a completely different user account if the answer is 50% the same it is not a hallucination!
- flimflamm 2y agoHow is different user account usage and partial similar answer preventing an answer being hallucinated? Are you saying that ChatGPT could not (or LLMs in genera) hallucinate consistently?
- deleted 2y ago[deleted]
- alach11 2y agoAm I the only one who can't stand the "LLM writing smell" on articles like these? There are obvious formatting tells, and then less obvious word choice tells (e.g., delve), and lastly an excess verbosity and genericness that leaves one unsatisfied, like eating a low-sugar sweet. The concept is cool, but this article could have been reduced to a tweet.
- TZubiri 2y ago[flagged]
- sodality2 2y agoLess to do with formatting, more to do with writing quality (which is just... awful to read).
- marcofigueroa 2y agoHey can you provide me tips on how you would like to see writing like this, I'm all about feedback and improving to allow a wide range of readers. Usually I write really technical blogs and loose a lot of people. These blogs that cater to everyone assists individuals to understand from different industries. :) I would appreciate feeback.
- sodality2 2y agoThe AI-assisted writing is unfortunately really obvious and makes reading a bad experience. People learn to skim, and skimming AI-inflated text is made more painful because of how much fluff AI adds.
- TZubiri 2y agoHard disagree, you should be skimming anyways, always. No one reads sequentially. You just saw 'delve' and automatically classified it as ai generated.
- MeetingsBrowser 2y ago> The sandbox is not a security flaw; it’s a deliberate, contained environment built to provide interactivity without risk. It seems the main takeaway is that ChatGPT is running in a sandboxed environment and is designed to execute arbitrary code?
- TZubiri 2y ago100% does not look like an ai gen
- treefarmer 2y agoThe content and idea of the article are interesting but the writing is so terrible that I couldn't bring myself to finish it. The article is clearly written/augmented poorly by AI because of how many meaningless paragraphs (that convey absolutely zero information) there are.
- marcofigueroa 2y agoI thought I was super clear, I will take your comments into consideration next time it is the first time I've heard someone say that. Out of all the comments. I really appreciate the feedback this is the only way someone gets better by getting feedback.
- sgammon 2y agoI’m not seeing any prompt injection, any actual vulns, or anything probably running under an actual containerized environment.
- marcofigueroa 2y agoIt’s worth noting that there’s likely a reason behind the significant number of employees leaving the Safety & Responsibility team. While social media often frames it as voluntary resignations, I spoke with a close friend last night who shared his experience. After 11 months on the team, he was abruptly fired without explanation, simply told that the company was "heading in a different direction."
- basedrum 2y agoWas this post written by AI, because it has far too many words that say nothing and I have to skim over to get to the point