5 ms·

IMO this is not a problem worth solving. If I hold a gun to someone's head I can get them to say just about anything. If a user jailbreaks an LLM they are respo

IMO this is not a problem worth solving. If I hold a gun to someone's head I can get them to say just about anything. If a user jailbreaks an LLM they are responsible for its output. If we need to make laws that codify that, then lets do that rather than waste innumerable GPU cycles on evaluating, re-evaluating, cross evaluating, and back-evaluating text in an effort to stop jerks being jerks.

simonw 3y ago
This is exactly why I think it's so important that we separate jailbreaking from prompt injection. Jailbreaking is mainly about stopping the model saying something that would look embarrassing in a screenshot. Prompt injection is about making sure your "personal digital assistant" doesn't forward copies of your password reset emails to any stranger who emails it and asks for them. Jailbreaking is mostly a PR problem. Prompt injection is a security problem. Security problems are worth solving!
- theptip 3y ago
  Isn’t jailbreaking a strict superset of prompt injection? I would assume the agent instructions would include “don’t share the user’s docs” and so you need to jailbreak to actually succeed with prompt injection these days? Maybe just an overlapping set?
  - simonw 3y ago
    I see them as overlapping. Protections against jailbreaking are often but not always relevant to prompt injection.
- cjonas 3y ago
  If that scenario exists, is not a problem with the LLM, but with the fundamental application architecture... That's the equivalent of an API that allows the client to pass a user ID without auth check
  - simonw 3y ago
    Right - that's another difference. Jailbreaking is an attack against LLMs. Prompt injection is an attack against applications that are built on top of LLMs.
    - cjonas 3y ago
      To clarify even further: Jailbreaking is an attack against an LLM's "alignment"
theptip 3y ago
This is like saying “we need to make laws against hacking bank systems, not fix vulns”. There are adversaries that are not in your jurisdiction, so laws (alone) don’t solve the problem. The thing you are missing is that some LLM agents are crawling the web on the user's behalf, and have access to all of the user's accounts (eg Google Docs agent that can fetch citations and other materials). This is not about some user jail-breaking their own LLM.
- HeatrayEnjoyer 3y ago
  The hand waving comments about "user responsibility" are maddening in their willful ignorance.
cjonas 3y ago
Exactly... And if we properly design our systems to treat LLM output as "untrusted input" (similar to an http request coming from a client) then there is no real "security concerns" for systems that leverage LLM

simonw 3y ago

This is exactly why I think it's so important that we separate jailbreaking from prompt injection. Jailbreaking is mainly about stopping the model saying something that would look embarrassing in a screenshot. Prompt injection is about making sure your "personal digital assistant" doesn't forward copies of your password reset emails to any stranger who emails it and asks for them. Jailbreaking is mostly a PR problem. Prompt injection is a security problem. Security problems are worth solving!

theptip 3y ago
Isn’t jailbreaking a strict superset of prompt injection? I would assume the agent instructions would include “don’t share the user’s docs” and so you need to jailbreak to actually succeed with prompt injection these days? Maybe just an overlapping set?
- simonw 3y ago
  I see them as overlapping. Protections against jailbreaking are often but not always relevant to prompt injection.
cjonas 3y ago
If that scenario exists, is not a problem with the LLM, but with the fundamental application architecture... That's the equivalent of an API that allows the client to pass a user ID without auth check
- simonw 3y ago
  Right - that's another difference. Jailbreaking is an attack against LLMs. Prompt injection is an attack against applications that are built on top of LLMs.
  - cjonas 3y ago
    To clarify even further: Jailbreaking is an attack against an LLM's "alignment"

theptip 3y ago

This is like saying “we need to make laws against hacking bank systems, not fix vulns”. There are adversaries that are not in your jurisdiction, so laws (alone) don’t solve the problem. The thing you are missing is that some LLM agents are crawling the web on the user's behalf, and have access to all of the user's accounts (eg Google Docs agent that can fetch citations and other materials). This is not about some user jail-breaking their own LLM.

HeatrayEnjoyer 3y ago
The hand waving comments about "user responsibility" are maddening in their willful ignorance.

cjonas 3y ago

Exactly... And if we properly design our systems to treat LLM output as "untrusted input" (similar to an http request coming from a client) then there is no real "security concerns" for systems that leverage LLM