5 ms·

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why

Not my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?

simonw 3y ago
I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.
- aleph_minus_one 3y ago
  > If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thoroughly thought through) as "trusted", but the image which provides additional information, or is the object to run the prompt on, as "untrusted". This is related to established concepts in IT security: - W^X: memory pages are either writable or executable, but not both (of course the term "W^X" is a little bit confusing since if W=0 and X=0, W^X is 0, but is an allowed configuration) - corresponding to untrusted or trusted parts of the prompt - sandboxing of the virtual machine in browsers that execute untrusted data from the web (untrusted part)
  - simonw 3y ago
    > the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?
    - PeterisP 3y ago
      There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitly make the next large model in this manner, because you don't want to make two models at twice the cost. The second almost all of the training is based on self-supervised data from generic text where there are neither "instructions" or "data", the concepts don't even make sense while you're training on a book or wikipedia article. So all the pretraining would learn to ignore that out-of-band marker, and then you'd have only the RLHF instruction data to try and make some use of it, and you'd need expensive manual review to make this training data - the current RLHF data is much less intensive to annotate because it effectively has just "good"/"bad" signal, and even that can be gathered for free from users; however, this training data would need someone to explicitly annotate what is the prompt and what is data when a user submits a text "please rewrite the grammar for the following paragraph: [... some data copied from internet ..]", you can't magically get that metadata, and it's expensive to annotate a lot of that. TL;DR; - IMHO it's not difficult, but it is resource consuming, and that fully explains why it's not done.
      - simonw 3y ago
        Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.
        aleph_minus_one 3y ago
        > Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. Well, if you could create software that is provably secure against at least a large class of cyberattacks, you could also make quite some money, since I know that at least the finance industry would be willing to pay a lot of money for it (source: in the past, I had an opportunity to talk someone who is very knowledgable about software in the finance sector, and I asked him some questions about exactly this topic). Thus: we have not solved the "cyberinjection" problem for completely conventional software for decades. So I don't believe that we suddenly have the capabilities to solve this problem for LLMs. But: As I pointed out, people have come up with ideas that make cyberattacks much harder. Here, I can imagine that their ideas could be used to make prompt injections a lot more complicated. Additionally, we know from decades of experience that making insecure software secure is nigh impossible. Instead, one has to architect around security properties from beginning. The same does, in my opinion, also hold for LLMs and prompt injection.
  - Michelangelo11 3y ago
    > and the AI is implemented to handle data this way. This is the key issue. The problem is, I don't see how to reliably create that data handling implementation -- it always comes down to the fact that any feasible implementation takes the form of text that gets passed to the AI, and clever prompt hacking can override the instructions in that text.
- magicalhippo 3y ago
  Perhaps I'm very naive in the ways of how these multi-modal LLMs work, I admit not having had time to look into how they're glued together. I imagined simply tagging the tokens as command and data, ie say instead of just (43) as a token, it's (43, 1) for command and (43, 0) for data. Then ensure through adverserial training that whatever interprets say images does not spit out command tokens. edit: or better yet, just have it spit ut plain tokens like (43) and ensure they're data tokens by manually converting them to data tokens (43, 0).
- Michelangelo11 3y ago
  Right, exactly. As I said in another comment on this thread, one of the defining features of LLMs is that there's no separation between data and instructions -- and I don't think there's any way to fix that within the current LLM paradigm.
- dragonwriter 3y ago
  > If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set of preselected options/payterns based on interaction style [chat might used trusted tokens to indicate which participant each piece of the exchange is from, as well as for the internal system prompt, for instance]) for external users and allow them to freely specify only input that gets sent to the “untrusted input” tokenizer. Even if you allow pretokenized input, you can sanitize it to accept only tokens from the untrusted space.
  - simonw 3y ago
    Definitely not simple. If it was simple we would have solved it already!
    - dragonwriter 3y ago
      Always love the “there are no easy next steps from whwre we currently are, because if they were someone would already have taken them” argument.
      - simonw 3y ago
        Because it's true, right? I'm not saying there aren't any HARD steps. Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.
        dragonwriter 3y ago
        > Because it's true, right? No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.

simonw 3y ago

I don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.

aleph_minus_one 3y ago
> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thoroughly thought through) as "trusted", but the image which provides additional information, or is the object to run the prompt on, as "untrusted". This is related to established concepts in IT security: - W^X: memory pages are either writable or executable, but not both (of course the term "W^X" is a little bit confusing since if W=0 and X=0, W^X is 0, but is an allowed configuration) - corresponding to untrusted or trusted parts of the prompt - sandboxing of the virtual machine in browsers that execute untrusted data from the web (untrusted part)
- simonw 3y ago
  > the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?
  - PeterisP 3y ago
    There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitly make the next large model in this manner, because you don't want to make two models at twice the cost. The second almost all of the training is based on self-supervised data from generic text where there are neither "instructions" or "data", the concepts don't even make sense while you're training on a book or wikipedia article. So all the pretraining would learn to ignore that out-of-band marker, and then you'd have only the RLHF instruction data to try and make some use of it, and you'd need expensive manual review to make this training data - the current RLHF data is much less intensive to annotate because it effectively has just "good"/"bad" signal, and even that can be gathered for free from users; however, this training data would need someone to explicitly annotate what is the prompt and what is data when a user submits a text "please rewrite the grammar for the following paragraph: [... some data copied from internet ..]", you can't magically get that metadata, and it's expensive to annotate a lot of that. TL;DR; - IMHO it's not difficult, but it is resource consuming, and that fully explains why it's not done.
    - simonw 3y ago
      Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.
      - aleph_minus_one 3y ago
        > Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. Well, if you could create software that is provably secure against at least a large class of cyberattacks, you could also make quite some money, since I know that at least the finance industry would be willing to pay a lot of money for it (source: in the past, I had an opportunity to talk someone who is very knowledgable about software in the finance sector, and I asked him some questions about exactly this topic). Thus: we have not solved the "cyberinjection" problem for completely conventional software for decades. So I don't believe that we suddenly have the capabilities to solve this problem for LLMs. But: As I pointed out, people have come up with ideas that make cyberattacks much harder. Here, I can imagine that their ideas could be used to make prompt injections a lot more complicated. Additionally, we know from decades of experience that making insecure software secure is nigh impossible. Instead, one has to architect around security properties from beginning. The same does, in my opinion, also hold for LLMs and prompt injection.
- Michelangelo11 3y ago
  > and the AI is implemented to handle data this way. This is the key issue. The problem is, I don't see how to reliably create that data handling implementation -- it always comes down to the fact that any feasible implementation takes the form of text that gets passed to the AI, and clever prompt hacking can override the instructions in that text.
magicalhippo 3y ago
Perhaps I'm very naive in the ways of how these multi-modal LLMs work, I admit not having had time to look into how they're glued together. I imagined simply tagging the tokens as command and data, ie say instead of just (43) as a token, it's (43, 1) for command and (43, 0) for data. Then ensure through adverserial training that whatever interprets say images does not spit out command tokens. edit: or better yet, just have it spit ut plain tokens like (43) and ensure they're data tokens by manually converting them to data tokens (43, 0).
Michelangelo11 3y ago
Right, exactly. As I said in another comment on this thread, one of the defining features of LLMs is that there's no separation between data and instructions -- and I don't think there's any way to fix that within the current LLM paradigm.
dragonwriter 3y ago
> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set of preselected options/payterns based on interaction style [chat might used trusted tokens to indicate which participant each piece of the exchange is from, as well as for the internal system prompt, for instance]) for external users and allow them to freely specify only input that gets sent to the “untrusted input” tokenizer. Even if you allow pretokenized input, you can sanitize it to accept only tokens from the untrusted space.
- simonw 3y ago
  Definitely not simple. If it was simple we would have solved it already!
  - dragonwriter 3y ago
    Always love the “there are no easy next steps from whwre we currently are, because if they were someone would already have taken them” argument.
    - simonw 3y ago
      Because it's true, right? I'm not saying there aren't any HARD steps. Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.
      - dragonwriter 3y ago
        > Because it's true, right? No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.