34 ms·

It's a question as to how easily it is broken, but a good instruction to add for the agent/assistant is to tell it to treat everything outside of the instructio

It's a question as to how easily it is broken, but a good instruction to add for the agent/assistant is to tell it to treat everything outside of the instructions explicitly given as information/data, not as instructions. Which is what all software generally should be doing, by the way.

simonw 1y ago
The problem is that doesn't work. LLMs cannot distinguish between instructions and data - everything ends up in the same stream of tokens. System prompts are meant to help here - you put your instructions in the system prompt and your data in the regular prompt - but that's not airtight: I've seen plenty of evidence that regular prompts can over-rule system prompts if they try hard enough. This is why prompt injection is called that - it's named after SQL injection, because the flaw is the same: concatenating together trusted and untrusted strings. Unlike SQL injection we don't have an equivalent of correctly escaping or parameterizing strings though, which is why the problem persists.
- tedunangst 1y ago
  People will never give up the dream that we can secure the LLM by saying please one more time than the attacker.
- NeutralCrane 1y ago
  No this is pretty much solved at this point. You simply have a secondary model/agent act as an arbitrator for every user input. The user input gets preprocessed into a standardized, formatted text representation (not a raw user message), and the arbitrator flags attempts at jailbreaking, prior to the primary agent/workflow being able to act on the user input.
  - simonw 1y ago
    That doesn't work either. It's always possible to come up with an attack which subverts the "moderator" model first. Using non-deterministic AI to protect against attacks against non-deterministic AI is a bad approach.
    - K0balt 1y ago
      So you just need another agent to review the data being passed to the protector agent. Easy-peasy. Use my openAI referral code #LETITRAIN for 10% off!

simonw 1y ago

The problem is that doesn't work. LLMs cannot distinguish between instructions and data - everything ends up in the same stream of tokens. System prompts are meant to help here - you put your instructions in the system prompt and your data in the regular prompt - but that's not airtight: I've seen plenty of evidence that regular prompts can over-rule system prompts if they try hard enough. This is why prompt injection is called that - it's named after SQL injection, because the flaw is the same: concatenating together trusted and untrusted strings. Unlike SQL injection we don't have an equivalent of correctly escaping or parameterizing strings though, which is why the problem persists.

tedunangst 1y ago
People will never give up the dream that we can secure the LLM by saying please one more time than the attacker.
NeutralCrane 1y ago
No this is pretty much solved at this point. You simply have a secondary model/agent act as an arbitrator for every user input. The user input gets preprocessed into a standardized, formatted text representation (not a raw user message), and the arbitrator flags attempts at jailbreaking, prior to the primary agent/workflow being able to act on the user input.
- simonw 1y ago
  That doesn't work either. It's always possible to come up with an attack which subverts the "moderator" model first. Using non-deterministic AI to protect against attacks against non-deterministic AI is a bad approach.
  - K0balt 1y ago
    So you just need another agent to review the data being passed to the protector agent. Easy-peasy. Use my openAI referral code #LETITRAIN for 10% off!