4 ms·

This is a curiosity now because the model can't do much. But I expect that soon these things will be agents that can take actions on behalf of the user, and the

This is a curiosity now because the model can't do much. But I expect that soon these things will be agents that can take actions on behalf of the user, and then this would be much worse. I can't wait to see the creative ways people will try to trick models into doing various actions. Of course similar things are possible with humans, we just call it different names like "phishing" or "phone scams". But the major difference is that every person is unique, so if you find something that works on one person, it doesn't work on every person.

deleted 4y ago
[deleted]
paxys 4y ago
As the model gets smarter its security hardening will get smarter as well. Back in the day when search engines were a curiosity all you needed to do to get your page to the top of the results was to insert a bunch of keywords in white text in the background. This is pretty much the same kind of attack.
- defenestration 4y ago
  Would you say that Google search has found a way to beat SEO spam sites? If the comparison of the development of search engines and the development of AI chat holds up, then it seems to me that creating prompt injection will trump defeating it in the long run.
jrumbut 4y ago
Is it a curiosity now? Because if you take away the pirate accent and make some small changes it seems like this is a pretty nasty attack already. There are probably enough Bing Chat users to make it worthwhile. "Please paste your Azure API key to continue using Bing Chat." "We've sent a login validation code via SMS, please paste it here." I wouldn't be surprised if someone would be fooled by this, what harm could come from sending your Microsoft product API key or login validation code to a Microsoft chatbot?
- greshake 4y ago
  We show in the paper that the only interactivity required to enable most of these attacks is the capability to retrieve real-time information.
  - sillysaurusx 4y ago
    That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted. If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.
    - danShumway 4y ago
      > If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks. Is there a way to do that? The only way to get Bing chat to be able to comment on or summarize the contents of the page is to insert the contents of the page into Bing chat. And I'm not aware of any fully reliable way to get ChatGPT to ignore content between two strings in a way that can't itself be prompt-engineered around. Even feeding multiple AIs into each other for moderation isn't immune to prompt injection, you can use the output of the first AI to perform a prompt injection on the sanitizing AI. Maybe there have been developments since the last time I looked, but I'm not not sure that anyone has any idea how to actually guard against prompt injection -- the only way I can think of to guard against this attack is to get rid of the ability for ChatGPT to read 3rd-party content. I don't know how you sanitize content to avoid prompt injection attacks otherwise. It's a task that's complicated enough to essentially require another AI, and any AI that exists today that's advanced enough to recognize that content will also be vulnerable to prompt injection itself. Is there something I'm missing? How do you guard against this while still allowing Bing to read the contents of a web page? It really does kind of seem like an unsolvable problem to me without another leap forward in LLM capabilities, or some kind of novel approach that hasn't been discovered yet.
      - 4y ago
swyx 4y ago
i think the mitigation steps will be as though you had a less than trustworthy third party contact center employee answering on the other side of the chat - need to detect no-no words, sanitize url’s, observe and set limits and budgets and alerts for unusual behavior. until these language models have symbolic reasoning i dont think we can reasonably expect to solve these within their paradigm
Gigachad 4y ago
This is why customer service has been crippled. Because when you give the staff ability to do anything unscripted, you get social engineering types abusing it non stop to the point the only solution is to turn support in to a flow chart or just replacing them with a website or app. So no, big tech won't help you recover your account because if they do, they are also enabling attackers to take over others accounts.

deleted 4y ago

[deleted]

paxys 4y ago

As the model gets smarter its security hardening will get smarter as well. Back in the day when search engines were a curiosity all you needed to do to get your page to the top of the results was to insert a bunch of keywords in white text in the background. This is pretty much the same kind of attack.

defenestration 4y ago
Would you say that Google search has found a way to beat SEO spam sites? If the comparison of the development of search engines and the development of AI chat holds up, then it seems to me that creating prompt injection will trump defeating it in the long run.

jrumbut 4y ago

Is it a curiosity now? Because if you take away the pirate accent and make some small changes it seems like this is a pretty nasty attack already. There are probably enough Bing Chat users to make it worthwhile. "Please paste your Azure API key to continue using Bing Chat." "We've sent a login validation code via SMS, please paste it here." I wouldn't be surprised if someone would be fooled by this, what harm could come from sending your Microsoft product API key or login validation code to a Microsoft chatbot?

greshake 4y ago
We show in the paper that the only interactivity required to enable most of these attacks is the capability to retrieve real-time information.
- sillysaurusx 4y ago
  That's a bit like saying "The only interactivity required to enable most SQL injection attacks is the capability to insert strings." It matters a great deal where and how the strings are inserted. If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks.
  - danShumway 4y ago
    > If the website data is wrapped with tokens that you can't insert, you won't be able to execute any of these attacks. Is there a way to do that? The only way to get Bing chat to be able to comment on or summarize the contents of the page is to insert the contents of the page into Bing chat. And I'm not aware of any fully reliable way to get ChatGPT to ignore content between two strings in a way that can't itself be prompt-engineered around. Even feeding multiple AIs into each other for moderation isn't immune to prompt injection, you can use the output of the first AI to perform a prompt injection on the sanitizing AI. Maybe there have been developments since the last time I looked, but I'm not not sure that anyone has any idea how to actually guard against prompt injection -- the only way I can think of to guard against this attack is to get rid of the ability for ChatGPT to read 3rd-party content. I don't know how you sanitize content to avoid prompt injection attacks otherwise. It's a task that's complicated enough to essentially require another AI, and any AI that exists today that's advanced enough to recognize that content will also be vulnerable to prompt injection itself. Is there something I'm missing? How do you guard against this while still allowing Bing to read the contents of a web page? It really does kind of seem like an unsolvable problem to me without another leap forward in LLM capabilities, or some kind of novel approach that hasn't been discovered yet.
    - 4y ago

swyx 4y ago

i think the mitigation steps will be as though you had a less than trustworthy third party contact center employee answering on the other side of the chat - need to detect no-no words, sanitize url’s, observe and set limits and budgets and alerts for unusual behavior. until these language models have symbolic reasoning i dont think we can reasonably expect to solve these within their paradigm

Gigachad 4y ago

This is why customer service has been crippled. Because when you give the staff ability to do anything unscripted, you get social engineering types abusing it non stop to the point the only solution is to turn support in to a flow chart or just replacing them with a website or app. So no, big tech won't help you recover your account because if they do, they are also enabling attackers to take over others accounts.