4 ms·
It seems to me like you'd need a separate security LLM context from the primary context that simply screens attempts to jailbreak out of system prompts. Somethi
by andrewmcwatters 2y ago
It seems to me like you'd need a separate security LLM context from the primary context that simply screens attempts to jailbreak out of system prompts. Something that simply categorizes attempts and then rejects the text from ever even making it to the primary context, like a sandbox.
But there are much more informed ML people out there than me, so I assume this and similar techniques have already been thought of.
- the-kenny 2y agoWith that, attackers need two gadgets instead of one. Where’s the end?
- doe_eyes 2y agoCommercial LLMs generally have input and output filters to prevent "bad" prompts from reaching the model (instead returning canned text), or to nuke output if it appears to violate certain criteria. But then, you have two independent mechanisms that can get out of sync, a classic source of issues in infosec - except both are also more or less inscrutable and fail in unexpected ways.
- slowmovintarget 2y agoNo. You need inference running in the user's session, sandboxed from inference running for some other user. Doubly so for RAG or agent control where the LLM is expected to operate knobs and levers with the borrowed authority of the user. The LLM is just generating orders of magnitude more user state as a result of a prompt. The actions the LLM is permitted to take must still be gated on the authorizations allowed for that user. This means that data unavailable to the user must not be in the training set of the LLM acting as an interface layer. The "we have to learn new lessons" only applies when you lump all data together and hope the LLM doesn't spit up someone else's data from a probabilistic elbow jog. Hope is not a strategy.