3 ms·

This is avoiding the core problem (mingling control and data) with security through obscurity. That can be an effective solution, but it's important to recogni

by fooker 3y ago

This is avoiding the core problem (mingling control and data) with security through obscurity. That can be an effective solution, but it's important to recognize it as such.

rst 3y ago
It's avoiding the problem by separating control and data, at unknown but signficant cost to functionality (the LLM which determines what tools get invoked doesn't see the actual data or results, only opaque tokens that refer to them, so it can't use them directly to make choices). I'm not sure how that qualifies as "security by obscurity".
- ChatGTP 3y ago
  I actually thought I read in his presentation that it’s probably not a great solution but better than nothing.
- fooker 3y ago
  It's attempting to split control and data through a system which is susceptible to the same issue. So prompt injection still works, you just have to find the right promt.
  - sdenton4 3y ago
    You're simply incorrect here: the point is that the quarantined LLM has no ability to execute code and all inputs and outputs are treated as untrusted strings. Thanks to the history of the Internet, handling untrusted strings is a thing we understand how to do. The privileged LLM doesn't see the untrusted text, and is prompted by the user - which is fine until the user does something dumb with the untrusted text. (Thus, the social engineering section.) Nothing about this is security by obscurity... It may be flawed ( feel free to provide an example that would cause a failure), but it's not just hiding a problem under a layer of rot13...
  - spion 3y ago
    The system as described is not susceptible to prompt injection: - The tool-using-LLM never sees data, only variables that are placeholders for the data. - A post tool-using-LLM templating layer translates variables into content before passing them to a concrete tool. - After variables are translated, only a non-priviledged (non-tool-using) LLM has access to the actual content. - The output of a non-priviledged LLM is again another variable represented e.g. by the tokens $OUTPUT. The tool LLM never sees into that content. It can give it to another tool, but it cannot see inside it. You can inject prompt into the non-priviledged LLM but it doesn't get to do anything.
  - jamilton 3y ago
    Prompt injection with this method could, at worst, make the plaintext incorrect. The summary could be replaced with spam, for example. Prompt injection with the naive method (just have 1 LLM doing everything) could, at worst, directly infect the user's computer.
phire 3y ago
I'm not sure it's possible to fix that "core problem". In the example of an AI assistant managing your emails, users want to be able to give it instructions like "delete that email about flowers" or "move all emails about the new house build to a folder". These control instructions are very context dependant on the data, and the LLM needs both to have any idea what to do about then.
JieJie 3y ago
I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt? Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?
- cwillu 3y ago
  A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A buffer overflow is fundamentally caused by a separation between allocation and use. A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts with a bunch of text, followed by more text, ultimately ending in (say) the original text repeated. Or transformed. Or whatever. The “emergent common-sense” that is the entire value of a language model is (I suspect) fundamentally in tension with providing restrictions on its output. We can bias the model, but there will always be _some_ weight for _any_ possible output, or else it wouldn't be possible to train the model in the first place.
  - cwillu 3y ago
    Which makes me wonder if there's any useful insight from a “0 and 1 are not probabilities” angle: the key being that some classes of output need to somehow be modified to actually have those “probabilities”.

by fooker 3y ago

This is avoiding the core problem (mingling control and data) with security through obscurity. That can be an effective solution, but it's important to recognize it as such.

rst 3y ago

It's avoiding the problem by separating control and data, at unknown but signficant cost to functionality (the LLM which determines what tools get invoked doesn't see the actual data or results, only opaque tokens that refer to them, so it can't use them directly to make choices). I'm not sure how that qualifies as "security by obscurity".

ChatGTP 3y ago
I actually thought I read in his presentation that it’s probably not a great solution but better than nothing.
fooker 3y ago
It's attempting to split control and data through a system which is susceptible to the same issue. So prompt injection still works, you just have to find the right promt.
- sdenton4 3y ago
  You're simply incorrect here: the point is that the quarantined LLM has no ability to execute code and all inputs and outputs are treated as untrusted strings. Thanks to the history of the Internet, handling untrusted strings is a thing we understand how to do. The privileged LLM doesn't see the untrusted text, and is prompted by the user - which is fine until the user does something dumb with the untrusted text. (Thus, the social engineering section.) Nothing about this is security by obscurity... It may be flawed ( feel free to provide an example that would cause a failure), but it's not just hiding a problem under a layer of rot13...
- spion 3y ago
  The system as described is not susceptible to prompt injection: - The tool-using-LLM never sees data, only variables that are placeholders for the data. - A post tool-using-LLM templating layer translates variables into content before passing them to a concrete tool. - After variables are translated, only a non-priviledged (non-tool-using) LLM has access to the actual content. - The output of a non-priviledged LLM is again another variable represented e.g. by the tokens $OUTPUT. The tool LLM never sees into that content. It can give it to another tool, but it cannot see inside it. You can inject prompt into the non-priviledged LLM but it doesn't get to do anything.
- jamilton 3y ago
  Prompt injection with this method could, at worst, make the plaintext incorrect. The summary could be replaced with spam, for example. Prompt injection with the naive method (just have 1 LLM doing everything) could, at worst, directly infect the user's computer.

phire 3y ago

I'm not sure it's possible to fix that "core problem". In the example of an AI assistant managing your emails, users want to be able to give it instructions like "delete that email about flowers" or "move all emails about the new house build to a folder". These control instructions are very context dependant on the data, and the LLM needs both to have any idea what to do about then.

JieJie 3y ago

I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt? Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?

cwillu 3y ago
A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A buffer overflow is fundamentally caused by a separation between allocation and use. A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts with a bunch of text, followed by more text, ultimately ending in (say) the original text repeated. Or transformed. Or whatever. The “emergent common-sense” that is the entire value of a language model is (I suspect) fundamentally in tension with providing restrictions on its output. We can bias the model, but there will always be _some_ weight for _any_ possible output, or else it wouldn't be possible to train the model in the first place.
- cwillu 3y ago
  Which makes me wonder if there's any useful insight from a “0 and 1 are not probabilities” angle: the key being that some classes of output need to somehow be modified to actually have those “probabilities”.