3 ms·

Sure but you've been here steadfast in your opinion that this is no big deal that is an easy fix away from being permanently resolved. It is not. It may be one

Sure but you've been here steadfast in your opinion that this is no big deal that is an easy fix away from being permanently resolved. It is not. It may be one of the hardest problems facing the deployment of these LLMs. "Sanitizing" these inputs when the language you are trying to parse is turing-complete is undecidable. It's a property that Rice's theorem applies to. I'll leave you with this quote of gwern: "... a language model is a Turing-complete weird machine running programs written in natural language; when you do retrieval, you are not 'plugging updated facts into your AI', you are actually downloading random new unsigned blobs of code from the Internet (many written by adversaries) and casually executing them on your LM with full privileges. This does not end well." - Gwern Branwen

sillysaurusx 4y ago
Would you like to bet money that 365 days from now, websites won’t be able to affect Bing the way that you’ve demonstrated in this PoC? I’ll happily take you up on whatever sum you choose. I didn’t say it was easy. I said it’s inevitable. There are straightforward ways to deal with this; all OpenAI + Microsoft needs to do is to choose one and implement it. Having a conversation with a user was also an undecidable task until one day it wasn’t. And the reason it became traceable is by using RL to reward the model for being conversational. It’s extremely straightforward to punish the model for misbehaving due to website injections, and the generalization of that is to punish the model for misbehaving due to text between two special BPE tokens (escaped text, I.e. website data). This is different than users being able to jailbreak chatgpt or Bing with prompts. When the user is prompting, they’re programming the model. So I agree that they won’t be able to defend against DAN attacks very easily without compromising the model’s performance in other areas. But that’s entirely different from sanitizing website data that Bing is merely looking at; such data can be trivially escaped with BPE tokens and RLHF will do the rest. If you do want to take me up on that bet, feel free to DM me on Twitter and we can hammer out the details. I’ll go any amount from $5 to $5k. Note that I’m not claiming that it’ll be impossible to craft a website that makes Bing go haywire, just that it’ll be so uncommon as to be pretty much impossible in practice, the same way that SQL injection attacks against AWS are rare but technically not impossible. We’ll hear about them as a CVE, Microsoft will fix the CVE, and life moves on, just like today with every other type of attack. The bet is that there are straightforward, quick (< 1 week) fixes for these problems, 365 days from today.

sillysaurusx 4y ago

Would you like to bet money that 365 days from now, websites won’t be able to affect Bing the way that you’ve demonstrated in this PoC? I’ll happily take you up on whatever sum you choose. I didn’t say it was easy. I said it’s inevitable. There are straightforward ways to deal with this; all OpenAI + Microsoft needs to do is to choose one and implement it. Having a conversation with a user was also an undecidable task until one day it wasn’t. And the reason it became traceable is by using RL to reward the model for being conversational. It’s extremely straightforward to punish the model for misbehaving due to website injections, and the generalization of that is to punish the model for misbehaving due to text between two special BPE tokens (escaped text, I.e. website data). This is different than users being able to jailbreak chatgpt or Bing with prompts. When the user is prompting, they’re programming the model. So I agree that they won’t be able to defend against DAN attacks very easily without compromising the model’s performance in other areas. But that’s entirely different from sanitizing website data that Bing is merely looking at; such data can be trivially escaped with BPE tokens and RLHF will do the rest. If you do want to take me up on that bet, feel free to DM me on Twitter and we can hammer out the details. I’ll go any amount from $5 to $5k. Note that I’m not claiming that it’ll be impossible to craft a website that makes Bing go haywire, just that it’ll be so uncommon as to be pretty much impossible in practice, the same way that SQL injection attacks against AWS are rare but technically not impossible. We’ll hear about them as a CVE, Microsoft will fix the CVE, and life moves on, just like today with every other type of attack. The bet is that there are straightforward, quick (< 1 week) fixes for these problems, 365 days from today.