4 ms·
Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and
by danpalmer 1y ago
Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees.
That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that your users won't click on a phishing email – you can train them, you can minimise risk, but ultimately you have to have a range of solutions applied together and some amount of trust. If we think about LLMs this way I think the conversation around security will be much more productive.
- LegionMammal978 1y agoThe thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
- thaumasiotes 1y ago> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.
- LegionMammal978 1y agoPerhaps not, but the same input will lead to the same distribution of outputs, so all an attacker has to do is design something that works with reasonable probability on their end, and everyone else's instances of the LLM will automatically be vulnerable. The same way a pest or disease can devastate a population of cloned plants, even if each one grows slightly differently.
- thaumasiotes 1y agoOK, but that's also the way attacking a bunch of individuals who can get tricked works.
- zwnow 1y agoFor tricking individuals your first got to contact them somehow. To trick an LLM you can just spam prompts.
- thaumasiotes 1y agoYou email them. It's called phishing.
- throwaway314155 1y agoRight and now there's a new vector for an old concept.
- zwnow 1y agoEmployees usually know to not click on random shit they get sent. Most mails alrdy get filtered before they even reach the employee. Good luck actually achieving something with phishing mails.
- thaumasiotes 1y agoWhen I was at NCC Group, we had a policy about phishing in penetration tests. The policy was "we'll do it if the customer asks for it, but we don't recommend it, because the success rate is 100%".
- bluefirebrand 1y agoHow can you ever get that lower than 100% if you don't do the test to identify which employees need to be trained / monitored because they fall for phishing?
- Retr0id 1y agoYou can still experimentally determine a strategy that works x% of the time, against a particular model. And you can keep refining it "offline" until x=99. (where "offline" just means invisible to the victim, not necessarily a local model)
- 33hsiidhkl 1y agoIt absolutely is deterministic, for any given seed value. Same seed = same output, every time, which is by definition deterministic.
- tough 1y agoonly if temperature is 0, but are they truly determinstic? I thought transformer based llm's where not
- 33hsiidhkl 1y agotemperature does not affect token prediction in the way you think. The seed value is still the seed value, before temperature calculations are performed. The randomness of an LLM is not related to its temperature. The seed value is what determines the output. For a specific seed value, say 42069, the LLM will always generate the same output, given the same input, given the same temperature.
- tough 1y agoThank you, I thought this wasn't the case (like it is with diffusion image models) TIL