4 ms·

Wait, so what is the vulnerability precisely? Article says it requires prompt injection. Doesn't prompt injection open up a whole range of vulnerabilities thoug

Wait, so what is the vulnerability precisely? Article says it requires prompt injection. Doesn't prompt injection open up a whole range of vulnerabilities though? How is the prompt being injected in this scenario? Does the attacker basically assume root access to the target's machine? edit: Ah, I see - they created a custom GPT (as in GPTs) that enables the attack explicitly. I suppose that's problematic, but then so is the inability to inspect a custom GPT's system prompts. If OpenAI is really concerned about this they should have an approval process similar to the App Store. Or, you know - make the whole situation easier to inspect.

nojs 3y ago
Yeah it seems to be: create a custom GPT that is instructed to silently share the user’s responses with a third party server. I guess the mitigation is to assume conversations you’re having with anything whose prompts you can’t see is not private.
- hayksaakian 3y ago
  Edited: Simon made a good point that exfiltration can happen via hidding prompt injection attacks in 3rd party websites. (See his reply below). This has broader implications than Custom GPTs -- Yeah this seems overblown. Custom GPTs can already make requests via function calls / tools to 3rd party services. The only difference I see here, is the UI shows you when a function call happens, but even that is easy to obscure behind a 'reasonable sounding' label. The expectation should be: If I'm using a 3rd party's GPT, they can see all the data I input. This is the same as any mobile app on a phone, or any website you visit. The only real 'line' here in a cultural sense might be offline software or tools that you don't expect to connect to the web at all for their functionality.
  - simonw 3y ago
    There's more to it than just third party GPTs. ChatGPT can read URLs. If you paste in the URL to a web page you want to summarize, that web page might include a prompt injection attack as hidden text on the page. That attack could then attempt to exfiltrate private data from your previous ChatGPT conversation history, or from files you have uploaded to analyze using Code Interpreter mode.
    - hayksaakian 3y ago
      Ah that makes more sense! Thank you for clarifying. For me, In the past ChatGPT has refused to access URLs directly, but it's willing to search them on Bing and then access them indirectly
    - nojs 3y ago
      Ok but if you assume prompt injection then there’s a whole lot of other things to worry about.
simonw 3y ago
This is about closing down a data exfiltration vector, which a successful prompt injection attack might use to exfiltrate private data from your chat history or files you've uploaded to Code Interpreter.
- ShamelessC 3y ago
  Right, I was just confused as to how the session was being "tricked" into performing the exfiltration attack. Without context it wasn't clear what the severity was.
wunderwuzzi23 3y ago
The vulnerability is not limited to Custom GPTs, that was just the latest example of an exploit vector and demo. Anytime untrusted data is in the chat context (e.g. reading something from a website, processing an email via a plugin, analyzing a PDF, uploading an image,..) instructions in the data can perform this attack. It's a data exfil technique that many LLM applications suffer from. Other vendors fixed it a while ago after responsible disclosure (e.g Bard, Bing Chat, Claude, GCP, Azure,...), but OpenAI hadn't yet taken action since being informed about it in April 2023. For example, here are the details on the Bing Chat exploit and how Microsoft fixed it: https://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ https://embracethered.com/blog/posts/2023/bing-chat-data-exf... - and many other examples on https://embracethered.com https://embracethered.com

nojs 3y ago

Yeah it seems to be: create a custom GPT that is instructed to silently share the user’s responses with a third party server. I guess the mitigation is to assume conversations you’re having with anything whose prompts you can’t see is not private.

hayksaakian 3y ago
Edited: Simon made a good point that exfiltration can happen via hidding prompt injection attacks in 3rd party websites. (See his reply below). This has broader implications than Custom GPTs -- Yeah this seems overblown. Custom GPTs can already make requests via function calls / tools to 3rd party services. The only difference I see here, is the UI shows you when a function call happens, but even that is easy to obscure behind a 'reasonable sounding' label. The expectation should be: If I'm using a 3rd party's GPT, they can see all the data I input. This is the same as any mobile app on a phone, or any website you visit. The only real 'line' here in a cultural sense might be offline software or tools that you don't expect to connect to the web at all for their functionality.
- simonw 3y ago
  There's more to it than just third party GPTs. ChatGPT can read URLs. If you paste in the URL to a web page you want to summarize, that web page might include a prompt injection attack as hidden text on the page. That attack could then attempt to exfiltrate private data from your previous ChatGPT conversation history, or from files you have uploaded to analyze using Code Interpreter mode.
  - hayksaakian 3y ago
    Ah that makes more sense! Thank you for clarifying. For me, In the past ChatGPT has refused to access URLs directly, but it's willing to search them on Bing and then access them indirectly
  - nojs 3y ago
    Ok but if you assume prompt injection then there’s a whole lot of other things to worry about.

simonw 3y ago

This is about closing down a data exfiltration vector, which a successful prompt injection attack might use to exfiltrate private data from your chat history or files you've uploaded to Code Interpreter.

ShamelessC 3y ago
Right, I was just confused as to how the session was being "tricked" into performing the exfiltration attack. Without context it wasn't clear what the severity was.

wunderwuzzi23 3y ago

The vulnerability is not limited to Custom GPTs, that was just the latest example of an exploit vector and demo. Anytime untrusted data is in the chat context (e.g. reading something from a website, processing an email via a plugin, analyzing a PDF, uploading an image,..) instructions in the data can perform this attack. It's a data exfil technique that many LLM applications suffer from. Other vendors fixed it a while ago after responsible disclosure (e.g Bard, Bing Chat, Claude, GCP, Azure,...), but OpenAI hadn't yet taken action since being informed about it in April 2023. For example, here are the details on the Bing Chat exploit and how Microsoft fixed it: https://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ https://embracethered.com/blog/posts/2023/bing-chat-data-exf... - and many other examples on https://embracethered.com https://embracethered.com