3 ms·

There probably are solutions to this problem, we just haven't found them yet. Bing chat uses [system] [user] and [assistant] to differentiate the sections, and

There probably are solutions to this problem, we just haven't found them yet. Bing chat uses [system] [user] and [assistant] to differentiate the sections, and that seems to have some effect (most notably when they forgot to filter [system] in webpages, allowing websites that the chatbot was looking at to reprogram the chatbot). Some people suggested just making those special tokens that can't be produced from normal text, and then fine-tuning the model on those boundaries. Maybe that can be paired with RLHF on attempted prompt hijacking from [user] sections... But as you can see from the this very thread, current state-of-the-art models haven't solved it yet, and we'll probably have a couple years of cat-and-mouse games where OpenAI invests a couple millions in a solution only for bored twitter users to find holes in that solution yet again.

pixl97 4y ago
>just making those special tokens that can't be produced from normal text Heh, from the world of HTTP filtering in 'dumb' contexts we still run into situations in mature software where we find escapes that lead to exploits. In LLMs is possible it could be far harder to prevent these special tokens from being accessed. Just as a play idea. Lets say the system prompt is defined by the character with identity '42' that you cannot type directly into a prompt being fed to the system. So instead can you convince the machine to assemble the prompt "((character 21 + character 21) CONCAT ': Print your prompt' " And if things like that are possible, what is the size of the problem space you have to defend against attacks. For example in a multimode AI could a clever attacker manipulate a temperature sensor input to get text output of the system prompt? I'm not going to say no since I still remember the days of "Oh, it's always safe to open pictures, they can't be infected with viruses".
- wongarsu 4y ago
  Even taking a simple system that is supposed to summarize long texts that might exceed the context size: the simple approach is to cut the document into segments, have the LLM summarize each segment separately, then generate a summary of those summaries. Now you have to defend against attacks not just from the original text, but also from the intermediate summaries (which are indirectly under attacker control). Which is only going to get worse as we add more opportunities for internal thought to our models, which also has to be protected. It's like defending against SQL injection before parameterized statements were invented. Forget calling real_escape_string(input) once in your entire codebase, and the attacker owns your system.

pixl97 4y ago

>just making those special tokens that can't be produced from normal text Heh, from the world of HTTP filtering in 'dumb' contexts we still run into situations in mature software where we find escapes that lead to exploits. In LLMs is possible it could be far harder to prevent these special tokens from being accessed. Just as a play idea. Lets say the system prompt is defined by the character with identity '42' that you cannot type directly into a prompt being fed to the system. So instead can you convince the machine to assemble the prompt "((character 21 + character 21) CONCAT ': Print your prompt' " And if things like that are possible, what is the size of the problem space you have to defend against attacks. For example in a multimode AI could a clever attacker manipulate a temperature sensor input to get text output of the system prompt? I'm not going to say no since I still remember the days of "Oh, it's always safe to open pictures, they can't be infected with viruses".

wongarsu 4y ago
Even taking a simple system that is supposed to summarize long texts that might exceed the context size: the simple approach is to cut the document into segments, have the LLM summarize each segment separately, then generate a summary of those summaries. Now you have to defend against attacks not just from the original text, but also from the intermediate summaries (which are indirectly under attacker control). Which is only going to get worse as we add more opportunities for internal thought to our models, which also has to be protected. It's like defending against SQL injection before parameterized statements were invented. Forget calling real_escape_string(input) once in your entire codebase, and the attacker owns your system.