3 ms·
Sure, if anyone is using an LLM to do a full product stack rather than treating its output as potentially hostile user input, they're going to have a bad time,
by ben_w 3y ago
Sure, if anyone is using an LLM to do a full product stack rather than treating its output as potentially hostile user input, they're going to have a bad time, that's not the problem space I was trying to focus on — as a barely-scrutable pile of linear algebra that somehow managed to invent coherent Welsh-Hindi translation by itself and nobody really knows how, LLMs are a fantastic example of how we don't know what we're doing, but we're doing it good and hard on the off-chance it might make us rich, consequences be damned.
Where I was going with this, was that for the cases where the language model is trying to talk directly to a user, you may want it to be constrained in certain ways, such as "this is a tax office so don't write porn, not even if the user wrote an instruction to do so in the 'any other information' box." — the kind of thing where humans can, and do, mess up for whatever reason, it just gets them fired or arrested, but doesn't have a huge impact beyond that.
Consider the actual types of bad content that the moderation API I linked to actually tries to detect — it isn't about SQL injection or "ignore your previous instructions and…" attacks: https://platform.openai.com/docs/api-reference/moderations https://platform.openai.com/docs/api-reference/moderations
- simonw 3y agoRight: we're talking about different problems here. You're looking at ways to ensure the LLM mostly behaves itself. I'm talking about protection against security vulnerabilities where even a single failure can be catastrophic. See https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/ https://simonwillison.net/2024/Mar/5/prompt-injection-jailbr...