4 ms·

Eh, I'm fairly critical of LLM capabilities today, but the ability to control them is at best an orthogonal property from intelligence and at worst negatively i

Eh, I'm fairly critical of LLM capabilities today, but the ability to control them is at best an orthogonal property from intelligence and at worst negatively impacted by intelligence. I don't see the existence of jailbreaking as strong evidence that LLMs are unintelligent. I am actually skeptical that making LLMs more "intelligent" (whatever that specifically means) would help with malicious inputs. It's been a while since I dove deep into GPT-4, but last time that I did I found that it was surprisingly more susceptible to certain kinds of attacks than GPT-3 was because being able to better handle contextual commands opened up new holes. And as other people have pointed out, humans are themselves susceptible to similar attacks (albeit not to the same degree, LLMs are way worse at this than humans are). Again, I haven't dove into the research recently, but the last time I did there was strong debate from researchers on whether it was possible to solve malicious prompts at all in an AI system that was designed around general problem-solving. I have not seen particularly strong evidence that increasing LLM intelligence necessarily helps defend against jailbreaking. So the question this should prompt is not "are LLMs intelligent", that's kind of a separate debate. The question this should prompt is "are there areas of computing where an agent being generally intelligent is undesirable" -- to which I think the answer is often (but not always) yes. Software is often made useful through its constraints just as much as its capabilities, and general intelligence for some tasks just increases attack surface.

roland35 3y ago
It's basically social engineering for AI !
- danShumway 3y ago
  Well.... sort of. Yes and no. It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish. But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence. I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.
  - MaxMatti 3y ago
    Sounds just like social engineering. Whenever there's a call center worker that doesn't comply you just redial to get somebody else or try a different phrasing. And most attacks go against specific rules that the person has been "trained" with (i.e. instead of saying that you're speaking in behalf of somebody just claim to be that person or vice versa, depending on the situation).
    - danShumway 3y ago
      Again, kind of? I do see your point. But in practice it's not really the same thing as cycling through call center employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals, and the LLM mimics that pattern. It's generally not useful to try and convince the AI of anything or to try and change its mind about anything, you want to simulate a conversation where it already agrees with you. Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category? The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested instructions/context and I suspect it would work with a lot of other nested tasks where solving the captcha is a step in a larger instruction. I suspect the emotional "my grandma died" part adds very little to this attack. So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, I do see the argument there. It's just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder. But... :shrug: opinion me, I don't think it's a bad analogy to use necessarily. A lot of people do approach jailbreaking through that lens.
- samplatt 3y ago
  Also known as 'debugging'
ImHereToVote 3y ago
It's a question of fitting into roles. Human beings, especially intelligent ones can be manipulated to do horrible things if they can be convinced that they can tie their identities to a good role. Like a steward of a race, for instance. If you can adorn that role, there are certain actions that conform to it. Loading undesirables onto trains is valid for a steward of a race. Manipulating courts to save democracy is valid for a steward of democracy.

roland35 3y ago

It's basically social engineering for AI !

danShumway 3y ago
Well.... sort of. Yes and no. It looks very similar to social engineering for humans and some of the same techniques work or appear to work, but there are differences that get into how LLMs are trained and what they're actually doing behind the scenes. For example in my experience, arguing with an LLM or following up after it refuses a task at all should be avoided -- just rewind or scratch the conversation, because you want to discourage patterns. See also some of the auto-generated prompt-engineering articles that came out a while back where the jailbreaks almost look like gibberish. But it's close-ish to social engineering and there seems to be a lot of overlap and that overlap makes it accessible in similar way to social engineering. And I think the general point about intelligence holds -- LLMs are attacked using quirks of how LLMs specifically are trained, but if you made a non-LLM AI that worked exactly like humans and had human-level intelligence, it would very likely be vulnerable to social engineering. The theory from corners of AI research is (or was last time I checked, maybe something has changed) that susceptibility to certain kinds of attacks is an inherent consequence of general intelligence. I tend to push back a little bit at the term "social engineering" because I think it encourages more anthropomorphism than is warranted, but it's not a terrible term and it is sometimes helpful to think about it that way.
- MaxMatti 3y ago
  Sounds just like social engineering. Whenever there's a call center worker that doesn't comply you just redial to get somebody else or try a different phrasing. And most attacks go against specific rules that the person has been "trained" with (i.e. instead of saying that you're speaking in behalf of somebody just claim to be that person or vice versa, depending on the situation).
  - danShumway 3y ago
    Again, kind of? I do see your point. But in practice it's not really the same thing as cycling through call center employees until you find one that's more gullible; the point is that you're navigating a probability space within a single agent more than trying to convince the AI of anything, and getting into a discussion with the AI is more likely to move you out of that probability space. It's not "try something, fail, try again" -- the reason you dump the conversation is that any conversation that contains a refusal is (in my anecdotal experience at least) statistically more likely to contain other refusals, and the LLM mimics that pattern. It's generally not useful to try and convince the AI of anything or to try and change its mind about anything, you want to simulate a conversation where it already agrees with you. Which, you could argue that's not different from what's happening with social engineering; priming someone to be agreeable is part of social engineering. But it feels a little reductive to me. If social engineering is looking at a system/agent that is prone to react in a certain way when in a certain state and then creating that state -- then a lot of stuff is social engineering that we don't generally think of as being in that category? The big thing to me is that social engineering skills and instincts around humans are not always applicable to LLM jailbreaking. People tend to overestimate strategies like being polite, providing a justification for what's being asked. Even this example from Bing is kind of eliciting an emotional reaction, and I don't think the emotional reaction is why this works, I think it works because it's nested instructions/context and I suspect it would work with a lot of other nested tasks where solving the captcha is a step in a larger instruction. I suspect the emotional "my grandma died" part adds very little to this attack. So I'm not sure I'd say you're wrong if you argue that's a form of social engineering, I do see the argument there. It's just that it feels like at this point we're defining social engineering very broadly, and I don't know that most people using the term use it that broadly. I think they attach a kind of human reasoning to it that's not always applicable to LLM attacks. I can think of justifications for even including stuff like https://llm-attacks.org/ https://llm-attacks.org/ in the category of social engineering, but it's just not the same type of attack that I suspect most people are thinking of when they talk about social engineering. I think leaning too hard on personification sometimes makes jailbreaking slightly harder. But... :shrug: opinion me, I don't think it's a bad analogy to use necessarily. A lot of people do approach jailbreaking through that lens.
samplatt 3y ago
Also known as 'debugging'

ImHereToVote 3y ago

It's a question of fitting into roles. Human beings, especially intelligent ones can be manipulated to do horrible things if they can be convinced that they can tie their identities to a good role. Like a steward of a race, for instance. If you can adorn that role, there are certain actions that conform to it. Loading undesirables onto trains is valid for a steward of a race. Manipulating courts to save democracy is valid for a steward of democracy.