4 ms·
It seems to me that the only robust solution has to be some sort of split-brain dual model where tainted data can only ever be input to a model which is only tr
by seanhunter 1y ago
It seems to me that the only robust solution has to be some sort of split-brain dual model where tainted data can only ever be input to a model which is only trained for sentence completion, not instruction-tuned.
Untainted data is the only data that can be input into the instruction-tuned half of the dual model.
In an architecture like this, any attempt to prompt inject would just find their injection harmlessly sentence-completed rather than turned into instructions and used to override other prompt instructions.
- wunderwuzzi23 1y agoYeah, improving robustness from prompt injection which such techniques will help. One attack avenue that is surprisingly not discussed much is that the model itself can be the attacker. In that case prompt injection is not the root cause, but a misaligned/backdoored model that might invoke tools is. So super risky use-cases should always require human oversight, but I'm worried we are already on a path of normalization of deviance. It's sort of the unlikely worse case scenario, but Murphys law reminds us that such an attack/accident will happen one day.