3 ms·
I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original ins
by JieJie 3y ago
I wonder if prompt injection is, at its core, is a buffer overflow error, where the buffer is the LLM's context. That it what is happening, no? The original instructions are overwritten by the injected prompt?
Would not, then, making adjustments to the context, either algorithmic, or by enlarging the context (100K Claude, perhaps?) go a long way towards solving the problem?
- cwillu 3y agoA buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A buffer overflow is fundamentally caused by a separation between allocation and use. A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts with a bunch of text, followed by more text, ultimately ending in (say) the original text repeated. Or transformed. Or whatever. The “emergent common-sense” that is the entire value of a language model is (I suspect) fundamentally in tension with providing restrictions on its output. We can bias the model, but there will always be _some_ weight for _any_ possible output, or else it wouldn't be possible to train the model in the first place.
- cwillu 3y agoWhich makes me wonder if there's any useful insight from a “0 and 1 are not probabilities” angle: the key being that some classes of output need to somehow be modified to actually have those “probabilities”.