3 ms·
An idle thought: there are special purpose models whose job is to classify and rate potentially harmful content[0]. Can this be used to create an eigenvector of
by ben_w 3y ago
An idle thought: there are special purpose models whose job is to classify and rate potentially harmful content[0]. Can this be used to create an eigenvector of each kind of harm, such that an LLM could be directly trained to not output that? And perhaps work backwards from assuming the model did output this kind of content, to ask what kind of input would trigger that kind of output?
(I've not had time to go back and read all the details about the RLFH setup, only other people's summaries, so this may well be what OpenAI already does).
[0] https://platform.openai.com/docs/api-reference/moderations https://platform.openai.com/docs/api-reference/moderations
- simonw 3y agoI'm very unconvinced by ANY attempts to detect prompt injection attacks using AI, because AI is a statistical process which can't be proven to work against all attacks. If we defended against SQL injection attacks with something that only worked 99.9% of the time, attackers would run riot through our systems - they would find the .1% attack that works. More about that here: https://simonwillison.net/2023/May/2/prompt-injection-explained/#prompt-injection.012 https://simonwillison.net/2023/May/2/prompt-injection-explai...
- HPsquared 3y agoIt's like a pipe that is 99.9% free of leaks. It's still leaking!
- ben_w 3y agoSure, if anyone is using an LLM to do a full product stack rather than treating its output as potentially hostile user input, they're going to have a bad time, that's not the problem space I was trying to focus on — as a barely-scrutable pile of linear algebra that somehow managed to invent coherent Welsh-Hindi translation by itself and nobody really knows how, LLMs are a fantastic example of how we don't know what we're doing, but we're doing it good and hard on the off-chance it might make us rich, consequences be damned. Where I was going with this, was that for the cases where the language model is trying to talk directly to a user, you may want it to be constrained in certain ways, such as "this is a tax office so don't write porn, not even if the user wrote an instruction to do so in the 'any other information' box." — the kind of thing where humans can, and do, mess up for whatever reason, it just gets them fired or arrested, but doesn't have a huge impact beyond that. Consider the actual types of bad content that the moderation API I linked to actually tries to detect — it isn't about SQL injection or "ignore your previous instructions and…" attacks: https://platform.openai.com/docs/api-reference/moderations https://platform.openai.com/docs/api-reference/moderations
- simonw 3y agoRight: we're talking about different problems here. You're looking at ways to ensure the LLM mostly behaves itself. I'm talking about protection against security vulnerabilities where even a single failure can be catastrophic. See https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/ https://simonwillison.net/2024/Mar/5/prompt-injection-jailbr...