A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A b

A buffer overflow is a useful reminder that security cuts through abstractions and needs to be built keeping in mind what its fundamentally being built on. A buffer overflow is fundamentally caused by a separation between allocation and use. A prompt “injection” is in a real sense a misnomer caused by forgetting what a completion engine does: a prompt can be “injected”, because there is a plausible text that starts with a bunch of text, followed by more text, ultimately ending in (say) the original text repeated. Or transformed. Or whatever. The “emergent common-sense” that is the entire value of a language model is (I suspect) fundamentally in tension with providing restrictions on its output. We can bias the model, but there will always be _some_ weight for _any_ possible output, or else it wouldn't be possible to train the model in the first place.