5 ms·

It's basically social engineering for AI !

danShumway 3y ago
Well.... sort of. Yes and no. It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish. But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence. I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.
- MaxMatti 3y ago
  Sounds just like social engineering. Whenever there's a call center worker that doesn't comply you just redial to get somebody else or try a different phrasing. And most attacks go against specific rules that the person has been "trained" with (i.e. instead of saying that you're speaking in behalf of somebody just claim to be that person or vice versa, depending on the situation).
  - danShumway 3y ago
    Again, kind of? I do see your point. But in practice it's not really the same thing as cycling through call center employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals, and the LLM mimics that pattern. It's generally not useful to try and convince the AI of anything or to try and change its mind about anything, you want to simulate a conversation where it already agrees with you. Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category? The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested instructions/context and I suspect it would work with a lot of other nested tasks where solving the captcha is a step in a larger instruction. I suspect the emotional "my grandma died" part adds very little to this attack. So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, I do see the argument there. It's just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder. But... :shrug: opinion me, I don't think it's a bad analogy to use necessarily. A lot of people do approach jailbreaking through that lens.
    - famouswaffles 3y ago
      >then a lot of stuff is social engineering that we don't generally think of as being in that category? I mean..yes? Social Engineering is just the malicious manifestation of general social navigation. I mean think about it. What's the actual difference between a child who waits until his mother is in a good mood to ask for sweets and a rogue agent who gets chatty with the security guard so he can be close by without seeming suspicious. It's not a difference of kind. It's purely intent. >Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works It is at the very least a big part of why. Appeal to emotion will consistently get better results regardless of task. https://arxiv.org/abs/2307.11760 https://arxiv.org/abs/2307.11760
      - danShumway 3y ago
        > I mean..yes? Social Engineering is just the malicious manifestation of general social navigation. I don't think "social" is the correct word to use alongside navigation in this sentence; an interaction with an LLM is not a social interaction. At least, if we classify it as a social interaction we might as well call credential stuffing or XSS attacks or buffer overflows a social interaction as well. Navigating a probabilistic space or a deterministic space is about as equivalent to social engineering as exploiting statistical flaws in an encryption algorithm is. Sure, you can make an argument that both of those things are similar to social engineering (and it might even be a convincing argument), but that's not really what people are thinking about when you use the word "social." The example you bring up is of a child and a parent, an extremely human example; your instinct is to think about this in human terms, not in a purely abstract "I am exploiting flaws in a semi-predictable system." So I still feel like there's some personification here that's not really accurate to what's going on during jailbreaking. LLMs do not have moods. Even starting from a premise that they're intelligent, they don't have a persistent identity, the most charitable interpretation of LLM intelligence and the most generous analysis of their capabilities would still call their internal experiences fundamentally alien to human experiences. The paper you link is interesting, I'll take a closer look at it. Without having taken the time to read through it fully, I don't know if I'd have any caveats to add, although it seems like a reasonable conclusion to me. We know that telling LLMs that they're experts can on its own produce better results in many cases. My own experience is that for jailbreaking emotion is a lot less valuable, but... :shrug: maybe there's a pattern there I didn't know how to take advantage of, I'm not going to disagree with the paper without reading it more closely. I will say that even taking the paper at face value, you have to ask: "is what's going on here actual emotional appeals to empathy or is it pattern-matching within a probability space for how conversations that include a plea for empathy are more likely to go?" I know that sounds like a pointless philosophical question, but it has really practical implications for how jailbreaking works because once you realize that it's all about pattern matching and probability and the emergent reasoning is part of that and feeds back into that, you realize that the attack surface is so much larger than just appeals to emotion or reasoning. In contrast though, if you're approaching jailbreaking as if you're talking to a human, then you're probably not using auto-generated jailbreaks because those don't look like human conversations, you're probably not using repetition as much as you should because excessive repetition would be bad to use when social engineering a human, you're probably not doing things like switching characters back and forth with the AI because nested roleplays or answering your own questions in the place of a target is not going to be very effective when trying to attack a human. Personification can lead to leaving tools on the table that (in my experience at least) are very effective at jailbreaking AIs and getting them to follow malicious prompts. There's a different way of approaching jailbreaking that doesn't make intuitive sense until you internalize "I am not talking to a human being and the same rules do not necessarily apply, even if they occasionally overlap."
samplatt 3y ago
Also known as 'debugging'

Well.... sort of. Yes and no. It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish. But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence. I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.

MaxMatti 3y ago
Sounds just like social engineering. Whenever there's a call center worker that doesn't comply you just redial to get somebody else or try a different phrasing. And most attacks go against specific rules that the person has been "trained" with (i.e. instead of saying that you're speaking in behalf of somebody just claim to be that person or vice versa, depending on the situation).
- danShumway 3y ago
  Again, kind of? I do see your point. But in practice it's not really the same thing as cycling through call center employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals, and the LLM mimics that pattern. It's generally not useful to try and convince the AI of anything or to try and change its mind about anything, you want to simulate a conversation where it already agrees with you. Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category? The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested instructions/context and I suspect it would work with a lot of other nested tasks where solving the captcha is a step in a larger instruction. I suspect the emotional "my grandma died" part adds very little to this attack. So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, I do see the argument there. It's just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder. But... :shrug: opinion me, I don't think it's a bad analogy to use necessarily. A lot of people do approach jailbreaking through that lens.
  - famouswaffles 3y ago
    >then a lot of stuff is social engineering that we don't generally think of as being in that category? I mean..yes? Social Engineering is just the malicious manifestation of general social navigation. I mean think about it. What's the actual difference between a child who waits until his mother is in a good mood to ask for sweets and a rogue agent who gets chatty with the security guard so he can be close by without seeming suspicious. It's not a difference of kind. It's purely intent. >Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works It is at the very least a big part of why. Appeal to emotion will consistently get better results regardless of task. https://arxiv.org/abs/2307.11760 https://arxiv.org/abs/2307.11760
    - danShumway 3y ago
      > I mean..yes? Social Engineering is just the malicious manifestation of general social navigation. I don't think "social" is the correct word to use alongside navigation in this sentence; an interaction with an LLM is not a social interaction. At least, if we classify it as a social interaction we might as well call credential stuffing or XSS attacks or buffer overflows a social interaction as well. Navigating a probabilistic space or a deterministic space is about as equivalent to social engineering as exploiting statistical flaws in an encryption algorithm is. Sure, you can make an argument that both of those things are similar to social engineering (and it might even be a convincing argument), but that's not really what people are thinking about when you use the word "social." The example you bring up is of a child and a parent, an extremely human example; your instinct is to think about this in human terms, not in a purely abstract "I am exploiting flaws in a semi-predictable system." So I still feel like there's some personification here that's not really accurate to what's going on during jailbreaking. LLMs do not have moods. Even starting from a premise that they're intelligent, they don't have a persistent identity, the most charitable interpretation of LLM intelligence and the most generous analysis of their capabilities would still call their internal experiences fundamentally alien to human experiences. The paper you link is interesting, I'll take a closer look at it. Without having taken the time to read through it fully, I don't know if I'd have any caveats to add, although it seems like a reasonable conclusion to me. We know that telling LLMs that they're experts can on its own produce better results in many cases. My own experience is that for jailbreaking emotion is a lot less valuable, but... :shrug: maybe there's a pattern there I didn't know how to take advantage of, I'm not going to disagree with the paper without reading it more closely. I will say that even taking the paper at face value, you have to ask: "is what's going on here actual emotional appeals to empathy or is it pattern-matching within a probability space for how conversations that include a plea for empathy are more likely to go?" I know that sounds like a pointless philosophical question, but it has really practical implications for how jailbreaking works because once you realize that it's all about pattern matching and probability and the emergent reasoning is part of that and feeds back into that, you realize that the attack surface is so much larger than just appeals to emotion or reasoning. In contrast though, if you're approaching jailbreaking as if you're talking to a human, then you're probably not using auto-generated jailbreaks because those don't look like human conversations, you're probably not using repetition as much as you should because excessive repetition would be bad to use when social engineering a human, you're probably not doing things like switching characters back and forth with the AI because nested roleplays or answering your own questions in the place of a target is not going to be very effective when trying to attack a human. Personification can lead to leaving tools on the table that (in my experience at least) are very effective at jailbreaking AIs and getting them to follow malicious prompts. There's a different way of approaching jailbreaking that doesn't make intuitive sense until you internalize "I am not talking to a human being and the same rules do not necessarily apply, even if they occasionally overlap."

samplatt 3y ago

Also known as 'debugging'