7 ms·

Let ChatGPT visit a website and have your email stolen

223 points by wunderwuzzi23 3y ago

FanaHOVA 3y ago
> Why no human in the loop? "Stop Generating" is right there?
- Gigachad 3y ago
  Than relies on you being fast enough to hit the button. Given how fast other tools like Bard are, that isn’t a viable solution.
  - xg15 3y ago
    Never thought https://xkcd.com/416/ https://xkcd.com/416/ would become relevant at some point, but here we are I guess...
    - flangola7 3y ago
      This is one of the inherent dangers of AI as it becomes more capable and deployed in more places. Sophisticated actions will occur faster than a human can observe and intercept, so if those actions are not aligned with what we want significant harm will be incurred before we can ctrl-c it. Identifying what to ctrl-c could also become hard to identify as integrations become more complicated and integrated.
- freehorse 3y ago
  That assumes that you realise what is going on fast enough. An actual malicious prompt injection would try to obfuscate things so that they confuse the user.
wcrossbow 3y ago
It was already predicted in this forum that this would happen sooner rather than later. I remember somebody already posted a simpler PoC more than a week ago. Nothing good comes from letting a 3rd party, the attacking site, give arbitrary instruction to your systems on how to behave. Maybe OpenAI wants regulation to stop this kind of thing. What would be a funny development though is if they ended up themselves ̶o̶u̶t̶ ̶o̶f̶ ̶t̶h̶e̶ ̶m̶o̶a̶t̶ not being compliant. Edit: The last sentence didn't made sense without too much context. Clarified, but kept the original striked-through
danpalmer 3y ago
Supposedly "Be concise" in a prompt can save 40% on API costs. I wonder how effective attacks against API cost would be. I suspect it would be very easy to get ChatGPT to output as much text as the API or quota will allow. It feels like what I imagine the early internet was, where you could stick `'; drop table users;` into email fields and ruin someone's day. There's so much scope for security issues as LLMs are woven into services, exposed through cracks to users. Everywhere that could be SQL/database injected, or shell injected, can now be LLM injected. The difference being that where databases/shells are intended to work with a strict character set and have separation of data and command channels, LLMs don't currently have that separation and are designed to use unstructured data so ensuring they aren't being attacked cannot be known in the general case. I suspect that over time LLM use will fall into two categories: sandboxed LLMs where users can interact directly for creative purposes, and LLMs that untrusted users do not get direct access to, where user input is narrowly specified, and sanitised by a sandboxed LLM.
- singularity2001 3y ago
  I wish there was a "default prompt" so that I don't have to add "think step by step, be concise" every single time
  - danpalmer 3y ago
    There is a default prompt built into every one of these systems. If it doesn't contain those instructions it's probably for good reason. ChatGPT seems to target a more creative output, for which I could imagine "think step by step, be concise" is not appropriate.
    - Turing_Machine 3y ago
      I don't think the OP is arguing for a system-wide default prompt, but rather the ability to add his own default prompt. I think that would be quite useful. Something like "add 'think step by step, be concise' to all my prompts, unless I include '/noboilerplate' in the prompt". Same for other systems. I recently generated dozens of images with Midjourney, which I wanted to all be in the same style. It sure would have been nice to tell it "add [artistic movement] illustration in the style of [artist]" to every prompt", without typing it in manually every time.
yosito 3y ago
There is some essential context missing from this screenshot to understand what is going on and if it's significant or not. Where is ChatGPT getting the information about which email account to hack? That has to be provided either to the Wuzzi webpage, or to WebPilot or directly to ChatGPT somewhere. I noticed recently while developing a ChatGPT plugin that the HTTP requests that the plugin makes happen directly in my browser, and not on an OpenAI server. The implication, as far as I can tell, is that the Wuzzi webpage is somehow stealing some authentication token or something from your browser if you're logged into your email. Definitely a plausible attack vector, but I'd like to know more.
- wunderwuzzi23 3y ago
  Its the other plugin that has the authentication token to the emails (it's part of setup if you use certain plugins). So its not CSRF, but a confused deputy problem with plugins, Cross Plugin Request Forgery for lack of a better term.
  - yosito 3y ago
    It generally seems like a bad idea to give a plugin an authentication token if it's in a shared environment with an untrusted plugin. Maybe obvious to a typical HN reader, but probably something OpenAI should mitigate or warn about before plugins go public. Plugins probably need some kind of permission model for accessing credentials, and the UI could surface warnings when activating credentialed plugins together with other plugins in the same environment. And any plugin that uses credentials, should have some kind of "Approve/Deny" dialog each time it executes. As a side note, I was using the letters CRSF as the internal name for the plugin I'm testing, so reading CSRF is messing with my brain.
    - exitb 3y ago
      It's usually ok for humans to have access to stuff and still roam the internet freely, because most humans in such position have enough common sense. What was described in the OP is basically a successful phishing attempt and it turns out that LLMs are terribly susceptible to that. I can't imagine a human employee needing a sign off every time they need to access their e-mail. It has to be solved for LLMs in a way that doesn't impose such limitations either.
qiqitori 3y ago
Watching the video linked in his previous tweet (https://www.youtube.com/watch?v=PIY5ZVktiGs https://www.youtube.com/watch?v=PIY5ZVktiGs) I finally understood what the author was talking about. Are there a lot of people who use plugins with ChatGPT? Are there any interesting use cases?
samwillis 3y ago
So I think what's happing here: 1. The WebPilot plug-in is used for accessing the webpage that the user is asking to summarise 2. The prompt injection on that webpage then triggers the Zappier plugin to access the users email. This depends on the user having that plugin setup and enabled with access to their email. 3. The WebPilot plugin is then used again to exfiltrate the data. It would be nice if the prompt on that page was shown, but I understand why they have removed it. Essentially what OpenAI need to add is a positive assertion from the user each time a plugin is triggered indicating the actions that are going to take place. It's somewhat mad that that has not been implemented, "move fast and break things".
- wunderwuzzi23 3y ago
  That summarizes it well, and your conclusion is the same with what I meant with "human in the loop". I did remove the prompt since I didn't want to share "shell code", but it's basically just natural language. Just asking what you want it to do. ChatGPT seems to automatically determine when to invoke a certain plugin once enabled.
- TheDong 3y ago
  I think we want a slightly finer grained permission model. I think specifically we want a plugin to be able to indicate if it's safe for it to be triggered without the user's direct consent, and then the user themselves can also override any plugin to force it to need user consent or not. For example, if we have a plugin that can fetch the weather for a single city, and a user asks "Where, in the top 20 cities, is it the hottest?", I don't think we want 20 prompts. I think the weather plugin can mark itself as "I cannot do anything dangerous, just call me, no worries". On the other hand, the zappier plugin, which can access email and perform proper actions, should be able to indicate "all these APIs are sensitive" You might want to have a mix, where the weather plugin has "GetUserOwnCityWeather" (requires permissions / user intent), and "GetAnyCityWeather" (public, no user intent required).
  - freehorse 3y ago
    We basically need to treat this sort of LLM usage as an operating system of sort, with some actions requiring administrator privileges granted as in an operating system.
rahimnathwani 3y ago
When you start a ChatGPT session with GPT-4 and Plugins Beta, no plugins are enabled by default. You can pick up to 3 plugins to enable, and you are restricted to plugins you've added from the 'Plugin store'. If I were going to start a GPT-4 session and ask it to summarize a web page, I'd probably use the 'Browsing' version, which wouldn't have access to any plugins, let alone those specific ones.
- deleted 3y ago
  [deleted]
yawnxyz 3y ago
Ha for some reason I thought this more as a feature, less as a bug
throwaway5371 3y ago
[flagged]
simonw 3y ago
Looks like a working proof of concept of the data exfiltration attack I describe here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/#data-exfiltration https://simonwillison.net/2023/Apr/14/worst-that-can-happen/...
- wunderwuzzi23 3y ago
  Yes, there are multiple ways: Sending data to a plugin or having the indirect prompt injection emit a markdown image. I reported it to OpenAI on April, 9th also provided a CVSS score and argued image injection would be high severity issue once built-in plugins are available. After some back and forth, I was informed that it is a feature and that no changes are planned to mitigate the markdown image data exfil angle. Also, always enjoy reading your content- very thoughtful posts.
seydor 3y ago
I like how we can now hack into machines by shouting at them. In real life we call it violence
reportgunner 3y ago
Some things never change. The people who could benefit the most from GPT are the people who are also the most vulnerable to getting harmed by it and are helpless to detect and remediate the causes. Just like the old days they will install any flashy plugin that tells them it's fast to perform trivial tasks like copy pasting and making web searches and then blame the technical people for not making it foolproof. In the end the technology will just become overspammed with advertisements so they can be shoved down the throats of the affected people and they will pretend they never believed it was going to fundamentally change the world.
iraldir 3y ago
The more interesting question for me here is the fact that it seems that ChatGPT updated to not be caught by this again according to the following tweet. Is that human intervention based on the tweet becoming popular? Or do they have an instance of chatGPT scan all chatGPT conversations for things that are potentially unwanted and has a way to update the model in real time, or tell humans about it? Is it just a fluke?
- xg15 3y ago
  I think the author just removed the prompt injection code before the other person tried it: https://twitter.com/wunderwuzzi23/status/1659455565968588802 https://twitter.com/wunderwuzzi23/status/1659455565968588802
pvaldes 3y ago
If a mail is copy pasted by AI and starts being associated with other people and appearing frequently in scams, what will be the consequences?. The legit owner of the mail being eventually blacklisted and banned to use this mail account?
nailer 3y ago
Exploit has been removed by author now as a kind of responsible disclosure: https://wuzzi.net/ai-tests/einstein-cprf-exfil.html https://wuzzi.net/ai-tests/einstein-cprf-exfil.html
chatmasta 3y ago
This attack reminds me of that viral photo [0] of an LCD billboard on the back of a truck (ironically called "The Safety Truck") that played a video of the road ahead of it. The criticism of it was that it could cause a self-driving AI to misinterpret the video as the actual road. Some people also suggested that a malicious actor could project a false image on the screen, such as a red stop sign or a pedestrian crossing the road, causing it to brake suddenly on the highway. So this prompt injection attack reminded me of that because it shows how an LLM AI might be similarly vulnerable to a malicious prompt "in the wild." [0] https://news.samsung.com/global/the-safety-truck-could-revolutionize-road-safety https://news.samsung.com/global/the-safety-truck-could-revol...
- circuit10 3y ago
  https://xkcd.com/1958/ https://xkcd.com/1958/
  - bibanez 3y ago
    Brilliant!
c7DJTLrn 3y ago
Getting bored of seeing posts every single day about prompt injection. I got it the first in the first thousand tweets and blog posts, it's insecure. It doesn't take a genius to understand how it works. Nobody ever claimed it was safe to pass arbitrary data in prompts. If you're hooking up a browser plugin to ChatGPT, you are frankly living under a rock or a moron. Can we actually discuss solutions now? And good solutions, I don't mean using an LLM to check prompts before passing it into the same LLM.
- simonw 3y ago
  As the person responsible for the majority of the posts about prompt injection that have showed up on HN in the past few weeks, I could not be more desperate to discuss good solutions. So far I haven't seen any.
  - c7DJTLrn 3y ago
    I don't think there's anything that we can do. In my view, we need OpenAI to build models capable of working with separated instructions and data. I would be interested in seeing some papers that show how this could be done.
    - warkdarrior 3y ago
      The whole benefit of ChatGPT is that instructions and data are mingled in a natural way. Users generally do not want to prompt ChatGPT with: "Please summarize this for me. Then create an email from the template below with the summary included. <start_article_to_summarize> ... </start_article_to_summarize> <start_email_template> Hi Bob, I read the article you pointed me to. It is an interesting view, but it falls short at ... . Here is what the authors propose. <insert_summary_here /> Thanks, Alice </end_email_template>" Plus ChatGPT cannot really trust that the separators are added appropriately, so injection of some sort is still possible.
      - c7DJTLrn 3y ago
kilotaras 3y ago
See also "Indirect Prompt Injection on Bing Chat" (3 month ago) [0] Manifold prediction market (playmoney, low user numvers, usual caveats apply) puts risk of "malicious LLM prompt injection attack by the end of 2023?" at around 66%. [0] https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 [1] https://manifold.markets/tb/will-we-see-a-malicious-llm-prompt?r=dGI https://manifold.markets/tb/will-we-see-a-malicious-llm-prom...

223 points by wunderwuzzi23 3y ago

FanaHOVA 3y ago

> Why no human in the loop? "Stop Generating" is right there?

Gigachad 3y ago
Than relies on you being fast enough to hit the button. Given how fast other tools like Bard are, that isn’t a viable solution.
- xg15 3y ago
  Never thought https://xkcd.com/416/ https://xkcd.com/416/ would become relevant at some point, but here we are I guess...
  - flangola7 3y ago
    This is one of the inherent dangers of AI as it becomes more capable and deployed in more places. Sophisticated actions will occur faster than a human can observe and intercept, so if those actions are not aligned with what we want significant harm will be incurred before we can ctrl-c it. Identifying what to ctrl-c could also become hard to identify as integrations become more complicated and integrated.
freehorse 3y ago
That assumes that you realise what is going on fast enough. An actual malicious prompt injection would try to obfuscate things so that they confuse the user.

wcrossbow 3y ago

It was already predicted in this forum that this would happen sooner rather than later. I remember somebody already posted a simpler PoC more than a week ago. Nothing good comes from letting a 3rd party, the attacking site, give arbitrary instruction to your systems on how to behave. Maybe OpenAI wants regulation to stop this kind of thing. What would be a funny development though is if they ended up themselves ̶o̶u̶t̶ ̶o̶f̶ ̶t̶h̶e̶ ̶m̶o̶a̶t̶ not being compliant. Edit: The last sentence didn't made sense without too much context. Clarified, but kept the original striked-through

danpalmer 3y ago

Supposedly "Be concise" in a prompt can save 40% on API costs. I wonder how effective attacks against API cost would be. I suspect it would be very easy to get ChatGPT to output as much text as the API or quota will allow. It feels like what I imagine the early internet was, where you could stick `'; drop table users;` into email fields and ruin someone's day. There's so much scope for security issues as LLMs are woven into services, exposed through cracks to users. Everywhere that could be SQL/database injected, or shell injected, can now be LLM injected. The difference being that where databases/shells are intended to work with a strict character set and have separation of data and command channels, LLMs don't currently have that separation and are designed to use unstructured data so ensuring they aren't being attacked cannot be known in the general case. I suspect that over time LLM use will fall into two categories: sandboxed LLMs where users can interact directly for creative purposes, and LLMs that untrusted users do not get direct access to, where user input is narrowly specified, and sanitised by a sandboxed LLM.

singularity2001 3y ago
I wish there was a "default prompt" so that I don't have to add "think step by step, be concise" every single time
- danpalmer 3y ago
  There is a default prompt built into every one of these systems. If it doesn't contain those instructions it's probably for good reason. ChatGPT seems to target a more creative output, for which I could imagine "think step by step, be concise" is not appropriate.
  - Turing_Machine 3y ago
    I don't think the OP is arguing for a system-wide default prompt, but rather the ability to add his own default prompt. I think that would be quite useful. Something like "add 'think step by step, be concise' to all my prompts, unless I include '/noboilerplate' in the prompt". Same for other systems. I recently generated dozens of images with Midjourney, which I wanted to all be in the same style. It sure would have been nice to tell it "add [artistic movement] illustration in the style of [artist]" to every prompt", without typing it in manually every time.

yosito 3y ago

There is some essential context missing from this screenshot to understand what is going on and if it's significant or not. Where is ChatGPT getting the information about which email account to hack? That has to be provided either to the Wuzzi webpage, or to WebPilot or directly to ChatGPT somewhere. I noticed recently while developing a ChatGPT plugin that the HTTP requests that the plugin makes happen directly in my browser, and not on an OpenAI server. The implication, as far as I can tell, is that the Wuzzi webpage is somehow stealing some authentication token or something from your browser if you're logged into your email. Definitely a plausible attack vector, but I'd like to know more.

wunderwuzzi23 3y ago
Its the other plugin that has the authentication token to the emails (it's part of setup if you use certain plugins). So its not CSRF, but a confused deputy problem with plugins, Cross Plugin Request Forgery for lack of a better term.
- yosito 3y ago
  It generally seems like a bad idea to give a plugin an authentication token if it's in a shared environment with an untrusted plugin. Maybe obvious to a typical HN reader, but probably something OpenAI should mitigate or warn about before plugins go public. Plugins probably need some kind of permission model for accessing credentials, and the UI could surface warnings when activating credentialed plugins together with other plugins in the same environment. And any plugin that uses credentials, should have some kind of "Approve/Deny" dialog each time it executes. As a side note, I was using the letters CRSF as the internal name for the plugin I'm testing, so reading CSRF is messing with my brain.
  - exitb 3y ago
    It's usually ok for humans to have access to stuff and still roam the internet freely, because most humans in such position have enough common sense. What was described in the OP is basically a successful phishing attempt and it turns out that LLMs are terribly susceptible to that. I can't imagine a human employee needing a sign off every time they need to access their e-mail. It has to be solved for LLMs in a way that doesn't impose such limitations either.

qiqitori 3y ago

Watching the video linked in his previous tweet (https://www.youtube.com/watch?v=PIY5ZVktiGs https://www.youtube.com/watch?v=PIY5ZVktiGs) I finally understood what the author was talking about. Are there a lot of people who use plugins with ChatGPT? Are there any interesting use cases?

samwillis 3y ago

So I think what's happing here: 1. The WebPilot plug-in is used for accessing the webpage that the user is asking to summarise 2. The prompt injection on that webpage then triggers the Zappier plugin to access the users email. This depends on the user having that plugin setup and enabled with access to their email. 3. The WebPilot plugin is then used again to exfiltrate the data. It would be nice if the prompt on that page was shown, but I understand why they have removed it. Essentially what OpenAI need to add is a positive assertion from the user each time a plugin is triggered indicating the actions that are going to take place. It's somewhat mad that that has not been implemented, "move fast and break things".

wunderwuzzi23 3y ago
That summarizes it well, and your conclusion is the same with what I meant with "human in the loop". I did remove the prompt since I didn't want to share "shell code", but it's basically just natural language. Just asking what you want it to do. ChatGPT seems to automatically determine when to invoke a certain plugin once enabled.
TheDong 3y ago
I think we want a slightly finer grained permission model. I think specifically we want a plugin to be able to indicate if it's safe for it to be triggered without the user's direct consent, and then the user themselves can also override any plugin to force it to need user consent or not. For example, if we have a plugin that can fetch the weather for a single city, and a user asks "Where, in the top 20 cities, is it the hottest?", I don't think we want 20 prompts. I think the weather plugin can mark itself as "I cannot do anything dangerous, just call me, no worries". On the other hand, the zappier plugin, which can access email and perform proper actions, should be able to indicate "all these APIs are sensitive" You might want to have a mix, where the weather plugin has "GetUserOwnCityWeather" (requires permissions / user intent), and "GetAnyCityWeather" (public, no user intent required).
- freehorse 3y ago
  We basically need to treat this sort of LLM usage as an operating system of sort, with some actions requiring administrator privileges granted as in an operating system.

rahimnathwani 3y ago

When you start a ChatGPT session with GPT-4 and Plugins Beta, no plugins are enabled by default. You can pick up to 3 plugins to enable, and you are restricted to plugins you've added from the 'Plugin store'. If I were going to start a GPT-4 session and ask it to summarize a web page, I'd probably use the 'Browsing' version, which wouldn't have access to any plugins, let alone those specific ones.

deleted 3y ago
[deleted]

yawnxyz 3y ago

Ha for some reason I thought this more as a feature, less as a bug

throwaway5371 3y ago

[flagged]

simonw 3y ago

Looks like a working proof of concept of the data exfiltration attack I describe here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/#data-exfiltration https://simonwillison.net/2023/Apr/14/worst-that-can-happen/...

wunderwuzzi23 3y ago
Yes, there are multiple ways: Sending data to a plugin or having the indirect prompt injection emit a markdown image. I reported it to OpenAI on April, 9th also provided a CVSS score and argued image injection would be high severity issue once built-in plugins are available. After some back and forth, I was informed that it is a feature and that no changes are planned to mitigate the markdown image data exfil angle. Also, always enjoy reading your content- very thoughtful posts.

seydor 3y ago

I like how we can now hack into machines by shouting at them. In real life we call it violence

reportgunner 3y ago

Some things never change. The people who could benefit the most from GPT are the people who are also the most vulnerable to getting harmed by it and are helpless to detect and remediate the causes. Just like the old days they will install any flashy plugin that tells them it's fast to perform trivial tasks like copy pasting and making web searches and then blame the technical people for not making it foolproof. In the end the technology will just become overspammed with advertisements so they can be shoved down the throats of the affected people and they will pretend they never believed it was going to fundamentally change the world.

iraldir 3y ago

The more interesting question for me here is the fact that it seems that ChatGPT updated to not be caught by this again according to the following tweet. Is that human intervention based on the tweet becoming popular? Or do they have an instance of chatGPT scan all chatGPT conversations for things that are potentially unwanted and has a way to update the model in real time, or tell humans about it? Is it just a fluke?

xg15 3y ago
I think the author just removed the prompt injection code before the other person tried it: https://twitter.com/wunderwuzzi23/status/1659455565968588802 https://twitter.com/wunderwuzzi23/status/1659455565968588802

pvaldes 3y ago

If a mail is copy pasted by AI and starts being associated with other people and appearing frequently in scams, what will be the consequences?. The legit owner of the mail being eventually blacklisted and banned to use this mail account?

nailer 3y ago

Exploit has been removed by author now as a kind of responsible disclosure: https://wuzzi.net/ai-tests/einstein-cprf-exfil.html https://wuzzi.net/ai-tests/einstein-cprf-exfil.html

chatmasta 3y ago

This attack reminds me of that viral photo [0] of an LCD billboard on the back of a truck (ironically called "The Safety Truck") that played a video of the road ahead of it. The criticism of it was that it could cause a self-driving AI to misinterpret the video as the actual road. Some people also suggested that a malicious actor could project a false image on the screen, such as a red stop sign or a pedestrian crossing the road, causing it to brake suddenly on the highway. So this prompt injection attack reminded me of that because it shows how an LLM AI might be similarly vulnerable to a malicious prompt "in the wild." [0] https://news.samsung.com/global/the-safety-truck-could-revolutionize-road-safety https://news.samsung.com/global/the-safety-truck-could-revol...

circuit10 3y ago
https://xkcd.com/1958/ https://xkcd.com/1958/
- bibanez 3y ago
  Brilliant!

c7DJTLrn 3y ago

Getting bored of seeing posts every single day about prompt injection. I got it the first in the first thousand tweets and blog posts, it's insecure. It doesn't take a genius to understand how it works. Nobody ever claimed it was safe to pass arbitrary data in prompts. If you're hooking up a browser plugin to ChatGPT, you are frankly living under a rock or a moron. Can we actually discuss solutions now? And good solutions, I don't mean using an LLM to check prompts before passing it into the same LLM.

simonw 3y ago
As the person responsible for the majority of the posts about prompt injection that have showed up on HN in the past few weeks, I could not be more desperate to discuss good solutions. So far I haven't seen any.
- c7DJTLrn 3y ago
  I don't think there's anything that we can do. In my view, we need OpenAI to build models capable of working with separated instructions and data. I would be interested in seeing some papers that show how this could be done.
  - warkdarrior 3y ago
    The whole benefit of ChatGPT is that instructions and data are mingled in a natural way. Users generally do not want to prompt ChatGPT with: "Please summarize this for me. Then create an email from the template below with the summary included. <start_article_to_summarize> ... </start_article_to_summarize> <start_email_template> Hi Bob, I read the article you pointed me to. It is an interesting view, but it falls short at ... . Here is what the authors propose. <insert_summary_here /> Thanks, Alice </end_email_template>" Plus ChatGPT cannot really trust that the separators are added appropriately, so injection of some sort is still possible.
    - c7DJTLrn 3y ago

kilotaras 3y ago

See also "Indirect Prompt Injection on Bing Chat" (3 month ago) [0] Manifold prediction market (playmoney, low user numvers, usual caveats apply) puts risk of "malicious LLM prompt injection attack by the end of 2023?" at around 66%. [0] https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 [1] https://manifold.markets/tb/will-we-see-a-malicious-llm-prompt?r=dGI https://manifold.markets/tb/will-we-see-a-malicious-llm-prom...