5 ms·

The Universal Interface currently has a huge problem in the form of "prompt injection" where currently we have no principled way to separate instructions and da

The Universal Interface currently has a huge problem in the form of "prompt injection" where currently we have no principled way to separate instructions and data, e.g. https://embracethered.com/blog/posts/2023/google-bard-data-exfiltration/ https://embracethered.com/blog/posts/2023/google-bard-data-e... Not to say this is the only problem, but this one is at least moderately well known and a clear and visible blocker.

menacingly 3y ago
I assume we're unlikely to truly mitigate anything with LLMs we can't mitigate in standard english.
londons_explore 3y ago
This would be easy enough to fix by having a modifier added to each token embedding vector to say "this is an instruction token" or "this is a data token".
- dragonwriter 3y ago
  It seems intuitively that that should work to mitigate some prompt injection, but even if it would, it’s quite possible that a good chunk of the power of modern models comes from not making that distinction, and the model in effect determining what kind of weight to give data by context. We might make much safer but also far less useful models by doing that. It also isn’t likely to eliminate all bad data –> undesirable actions within the scope of what is enabled by the tools the model has access to problems. Because as long as the intent is for behavior to depend on data, you can manipulate behavior by manipulating data in unexpected ways.
- intended 3y ago
  LLM complexity depends on its use case - if it’s a chat window, this kind of dichotomy wouldnt work (Unless I am missing something in this thread) The model will be treating all text as input, whether it’s tagged as instruct or data. You could find a way to abuse either entry point.
- Eridrus 3y ago
  I agree this is a natural thing to try. Given I have not seen anyone report success with this, I lean towards thinking this doesn't actually work reliably (or the entire community is asleep at the wheel).

menacingly 3y ago

I assume we're unlikely to truly mitigate anything with LLMs we can't mitigate in standard english.

londons_explore 3y ago

This would be easy enough to fix by having a modifier added to each token embedding vector to say "this is an instruction token" or "this is a data token".

dragonwriter 3y ago
It seems intuitively that that should work to mitigate some prompt injection, but even if it would, it’s quite possible that a good chunk of the power of modern models comes from not making that distinction, and the model in effect determining what kind of weight to give data by context. We might make much safer but also far less useful models by doing that. It also isn’t likely to eliminate all bad data –> undesirable actions within the scope of what is enabled by the tools the model has access to problems. Because as long as the intent is for behavior to depend on data, you can manipulate behavior by manipulating data in unexpected ways.
intended 3y ago
LLM complexity depends on its use case - if it’s a chat window, this kind of dichotomy wouldnt work (Unless I am missing something in this thread) The model will be treating all text as input, whether it’s tagged as instruct or data. You could find a way to abuse either entry point.
Eridrus 3y ago
I agree this is a natural thing to try. Given I have not seen anyone report success with this, I lean towards thinking this doesn't actually work reliably (or the entire community is asleep at the wheel).