3 ms·

Getting bored of seeing posts every single day about prompt injection. I got it the first in the first thousand tweets and blog posts, it's insecure. It doesn't

Getting bored of seeing posts every single day about prompt injection. I got it the first in the first thousand tweets and blog posts, it's insecure. It doesn't take a genius to understand how it works. Nobody ever claimed it was safe to pass arbitrary data in prompts. If you're hooking up a browser plugin to ChatGPT, you are frankly living under a rock or a moron. Can we actually discuss solutions now? And good solutions, I don't mean using an LLM to check prompts before passing it into the same LLM.

simonw 3y ago
As the person responsible for the majority of the posts about prompt injection that have showed up on HN in the past few weeks, I could not be more desperate to discuss good solutions. So far I haven't seen any.
- c7DJTLrn 3y ago
  I don't think there's anything that we can do. In my view, we need OpenAI to build models capable of working with separated instructions and data. I would be interested in seeing some papers that show how this could be done.
  - warkdarrior 3y ago
    The whole benefit of ChatGPT is that instructions and data are mingled in a natural way. Users generally do not want to prompt ChatGPT with: "Please summarize this for me. Then create an email from the template below with the summary included. <start_article_to_summarize> ... </start_article_to_summarize> <start_email_template> Hi Bob, I read the article you pointed me to. It is an interesting view, but it falls short at ... . Here is what the authors propose. <insert_summary_here /> Thanks, Alice </end_email_template>" Plus ChatGPT cannot really trust that the separators are added appropriately, so injection of some sort is still possible.
    - c7DJTLrn 3y ago
      Does it need to be either/or? We can have models for chat and models for APIs.
- Joeri 3y ago
  I would say: run with it. Assume the model's output is always untrusted. Assume that no rules can be applied to the model itself, only to the container it runs in. What kind of sandbox does the model need to be placed in given those assumptions? Simple rules can be followed like: never take a potentially harmful action based on the model's output unless the user has had the ability to review the action, clear the context and apply whitelist validation to the action's parameters before using the model itself to ask the user for confirmation, never put data in the context that you don't want the user to find out, etc... I think it's mostly a matter of UI design, not of prompt engineering.
crooked-v 3y ago
> It doesn't take a genius to understand how it works. Unfortunately, the bar is substantially lower than you think: https://www.rollingstone.com/culture/culture-features/texas-am-chatgpt-ai-professor-flunks-students-false-claims-1234736601/ https://www.rollingstone.com/culture/culture-features/texas-...
rain1 3y ago
There are no solutions
famouswaffles 3y ago
>I don't mean using an LLM to check prompts before passing it into the same LLM. I mean this works for GPT-4 lol. It's just twice as expensive

simonw 3y ago

As the person responsible for the majority of the posts about prompt injection that have showed up on HN in the past few weeks, I could not be more desperate to discuss good solutions. So far I haven't seen any.

c7DJTLrn 3y ago
I don't think there's anything that we can do. In my view, we need OpenAI to build models capable of working with separated instructions and data. I would be interested in seeing some papers that show how this could be done.
- warkdarrior 3y ago
  The whole benefit of ChatGPT is that instructions and data are mingled in a natural way. Users generally do not want to prompt ChatGPT with: "Please summarize this for me. Then create an email from the template below with the summary included. <start_article_to_summarize> ... </start_article_to_summarize> <start_email_template> Hi Bob, I read the article you pointed me to. It is an interesting view, but it falls short at ... . Here is what the authors propose. <insert_summary_here /> Thanks, Alice </end_email_template>" Plus ChatGPT cannot really trust that the separators are added appropriately, so injection of some sort is still possible.
  - c7DJTLrn 3y ago
    Does it need to be either/or? We can have models for chat and models for APIs.
Joeri 3y ago
I would say: run with it. Assume the model's output is always untrusted. Assume that no rules can be applied to the model itself, only to the container it runs in. What kind of sandbox does the model need to be placed in given those assumptions? Simple rules can be followed like: never take a potentially harmful action based on the model's output unless the user has had the ability to review the action, clear the context and apply whitelist validation to the action's parameters before using the model itself to ask the user for confirmation, never put data in the context that you don't want the user to find out, etc... I think it's mostly a matter of UI design, not of prompt engineering.

crooked-v 3y ago

> It doesn't take a genius to understand how it works. Unfortunately, the bar is substantially lower than you think: https://www.rollingstone.com/culture/culture-features/texas-am-chatgpt-ai-professor-flunks-students-false-claims-1234736601/ https://www.rollingstone.com/culture/culture-features/texas-...

rain1 3y ago

There are no solutions

famouswaffles 3y ago

>I don't mean using an LLM to check prompts before passing it into the same LLM. I mean this works for GPT-4 lol. It's just twice as expensive