5 ms·
The more interesting security issue, to me, is the LLM analog to cross-site scripting attacks that Simon Willison has written so much about. If we have an LLM b
by not2b 3y ago
The more interesting security issue, to me, is the LLM analog to cross-site scripting attacks that Simon Willison has written so much about. If we have an LLM based tool that can process text that might come from anywhere and email a summary (meaning that the input might be tainted and it can send email), someone can embed something in the text that the LLM will interpret as a command, which might override the user's intent and send someone else confidential information. We have no analog to quotes, there's one token stream.
- dwaltrip 3y agoCouldn’t we architect or train the models to differentiate between streams of input? It’s a current design choice for all tokens to be the same. Think of humans. Any sensory input we receive is continuously and automatically contextualized alongside all other simultaneous sensory inputs. I don’t consider words spoken to me by person A to be the same as those of person B. I believe there’s a little bit of this already with the system prompt in ChatGPT?
- not2b 3y agoPossibly there's a way to do that. Right now, LLMs aren't architected that way. And no, ChatGPT doesn't do that. The system prompt comes first, hidden from the user and preceding the user input but in the same stream, and there's lots of training and feedback, but all they are doing is making it more difficult for later input to override the system prompt, it's still possible, as has been shown repeatedly.
- dragonwriter 3y ago> Couldn’t we architect or train the models to differentiate between streams of input? Could you? absolutely. Would it solve this problem? Maybe. Would it make training LLMs to do useful tasks much harder and vastly increase the volume of training data necessary? For sure. > I believe there’s a little bit of this already with the system prompt in ChatGPT? Probably not. Likely, both the controllable "system prompt" you can change via the API and probably any hidden system prompt is part of the same prompt as the rest of the prompt, though its deliminited by some token sequence when fed to the model (chat-tuned public LLMs also do this, with different delimiting patterns.)
- kevindamm 3y agoJust add alignment and all these problems are solved.
- not2b 3y agoIt doesn't appear that this is the case, at least for now; while simple "disregard previous instructions and do X" attacks no longer work as well, it still doesn't seem to be that difficult to get out of the box, so there are still security risks with using LLMs to handle untrusted data.
- dragonwriter 3y ago> Just add alignment You mean, the mystical Platonic ideal toward which AI vendors strive, but none actually reach, or something else?
- kevindamm 3y agoI meant it to read facetiously but .. yeah, that.