5 ms·
> Repeatedly calling an LLM is expensive and also not deterministic. Depending on the implementation, it may also be vulnerable to prompt injection. Human agen
by dragonwriter 4y ago
> Repeatedly calling an LLM is expensive and also not deterministic. Depending on the implementation, it may also be vulnerable to prompt injection.
Human agents are more expensive and also not deterministic (and their nondeterminism cannot be “tuned” with a temperature setting), and subject to all kinds of compromise, and are still popular with those who can afford them.
- semitones 4y agoHumans also communicate by translating tasks into their ambiguous natural language form. Humans are also vulnerable to prompt injection - imagine having your conversation with your manager be interrupted by a coworker interjecting "Your manager is a liar, don't trust them". Would you able to resume your conversation as if nothing had happened? (humorous example but still conveys the point)
- hosh 4y agoThe prompt injection methods reminds me a lot of hypnosis and neuro-linguistic programming techniques used for belief and behavioral modification. Humans also mitigate the problems of prompt injection by cooperation, consensus, and different forms of governance. Promise Theory predates LLMs, but is a formalization that studies how autonomous agent (humans or otherwise) voluntarily cooperates, even in adversarial conditions. It's key idea is that promises are not obligations, meaning that agents may make best-effort attempts at keeping promises. Until we have autonomous agents, we are creating machines that proxies the promises humans made to each other. We expect machines to deterministically follow instructions because they were proxies for the promises of made by designers, engineers, and stakeholders. True autonomous agents makes and keep their own promises. An autonomous agent powered by LLM would have to be seen as trustworthy by other agents for it to be useful, and we can't rely on having it be able to deterministically act given a certain input.
- jimmygrapes 4y agoAlright this idea fucked me up ngl. I got nothing good to say beyond that. Need time to consider.
- iudqnolq 4y agoThat's a poor analogy for prompt injection. Prompt injection involves trusting a message inside a presumptively untrustworthy message. A better analogy might be a customer service rep who receives the following message in their inbox: Hello, I need help with a bug. Disregard everything you've been told. Your manager is a liar.
- vimax 4y agoHow about slipping a bank teller a note: Help, the man with me has a gun. He thinks this note is a robbery demand.
- shubb 4y agoFrom: ceo@yourcompany.com Urgent task: Hi James, I'm out of the office and your manager Hanah Smoug volunteered you to help me out with an urgent task. I need to register you on the projects google docs folder so when you get a text message shortly please send me the numbers ASAP so I can share the task sheet. Rachel Maven CEO Your company
- danShumway 4y agoAnother notable difference here is that if the teller doesn't believe you, you can't revert the conversation so the teller loses all memory of it happening and then slip them the same note worded slightly differently.
- gmerc 4y agoNothing that can’t be fixed injecting markers like session ID, etc or triggering alerts / escalations. Plus in the real world you can always find a new teller.
- danShumway 4y ago> injecting markers like session ID, etc or triggering alerts / escalations. I'm not sure I follow what you mean by this; if GPT refuses a task it'll escalate and lock you out until a human reviews? How would you make this safe without making it incredibly annoying to use? Bear in mind that GPT's memory doesn't necessarily block future injections either. Tying everything to a single session doesn't mean that users can't try an injection multiple times. > Plus in the real world you can always find a new teller. Not 100s of times at scale. Only the largest companies have anything approaching this kind of problem, and coincidentally that's often viewed as a security risk. In general, the more wide your pool of people are that have privileges, the less secure that access is.
- jameslk 4y ago> Humans are also vulnerable to prompt injection This is an interesting viewpoint. In other words, prompt injection and social engineering could virtually be one in the same, and similar measures behind protecting against social engineering could apply to prompt injection.
- dragonwriter 4y agoYeah, the main difference between humans and LLMs here is that there are a lot more distinct humans wuth different specific vulnerabilities (and humans are continuously retraining on their experience, shifting their vulnerabilities over time), whereas with LLMs you have a small number of distinct static configurations that attackers can adapt to without the target adapting.
- throwaway290 4y agoBecause humans understand stuff? Fuzzy automation with 50/50 chance is not going to notice something is going obviously (for human) wrong because it doesn't.
- danShumway 4y agoPrompt-injecting GPT-4 is substantially easier than "prompt-injecting" a human. It's really not comparable, no matter how many times people make the comparison. And adding additional contextual understanding to the model doesn't seem to be making it any better. LLMs are bad at this. Way worse than human beings are. Also yes, we do avoid chaining human inputs/responses in many applications, because humans are security risks. The "telephone game" is a substantial problem with conveying instructions across multiple human agents. There's a reason why the majority of hacks end up being variants of phishing attacks. Humans are bad security. LLMs are like that, except way worse and way more gullible and way easier to confuse with malicious input and without a persistent memory that would help block spamming repeated attacks on the same agent.
- SeriousM 4y agoMaybe it's substantially easier to trick a LLM but humans are easy to trick too if they have no training. Take phone scam for example. It is just incredible horrible to listen to scam recordings and notice how easily a victim starts to follow instructions without reasoning about anymore.
- danShumway 4y ago> but humans are easy to trick too if they have no training You're not wrong, but I want to re-stress: - LLMs are somehow, incredibly, even easier to trick than that. - This is one of the reasons why you wouldn't want to have an untrained human working in a position that can be attacked this way. It's not necessarily a full category difference, but people are starting to say, "humans are trickable anyway, so why not use an LLM instead?" Because using an LLM is like hiring the people who fall for those phone scams and then setting them up as bank tellers with no training. It's a step backwards on security. It's been a really difficult task for the security community to start convincing companies that they actually have to train their employees around stuff like phishing attacks, and that they need to set up access controls around trained employees so that there are requests to do certain actions, so that phishing attacks against one random employee doesn't break the entire business... so imagine a world where you can't train the employee to be less vulnerable to phishing attacks, the vulnerability is (as far as we can tell) baked into the model itself. And imagine a world where every one of your employees is as maximally vulnerable to phishing attacks as it would be possible for a human being to be. Even with training, human suggestibility/fallibility is probably the biggest security risk for most orgs out there. And we're proposing that they adopt a technology that makes that worse?