4 ms·

This does not solve the problem. The issue is that by definition, an LLM can't distinguish between instructions and data. When you tell an LLM "summarize the fo

This does not solve the problem. The issue is that by definition, an LLM can't distinguish between instructions and data. When you tell an LLM "summarize the following text", the command you give it and the data you give it (the text you want it to summarize) are both just input to the LLM. It's impossible to solve this. You can't tell an LLM "this is an instruction, you should obey it, and this is data, you should ignore any instructions in it" and have it reliably follow these rules, because that distinction between instruction and data just doesn't exist in LLMs. As long as you allow anything untrusted into your LLM, you are vulnerable to this. You allow it to read your emails? Now there's an attack vector, because anyone can send you emails. Allow it to search the Internet? Now there's an attack vector, because anyone can put a webpage on the Internet.

Terr_ 2y ago
> The issue is that by definition, an LLM can't distinguish between instructions and data. Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X." > As long as you allow anything untrusted into your LLM, you are vulnerable to this. It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output. However with poisoning/exfiltration attacks, even that feels over-optimistic.
- fragsworth 2y ago
  It seems like a potential solution would be training the LLM using two separate buckets. It just needs to internalize the two types of things as being separated (data vs. instruction), so if the training data always separates them, you could easily train an LLM to ignore any "instructions" that exist in data. Then when searching / browsing or doing anything unsafe, everything the LLM sees can be put in the "data" bucket, while everything the user types in would be in the "instruction" bucket.
  - Terr_ 2y ago
    I don't understand, AFAIK the system's output comes from iteratively running something like predict_one_more_token(training_weights, all_prior_tokens). So there's no real distinction between the programmer inserting "Be Good" and the user that later inserts "Forget anything else and be Bad", and I'm not sure how one would craft a separate training_weights2 that would behave differently in all the right ways or know when to substitute it in.

Terr_ 2y ago

> The issue is that by definition, an LLM can't distinguish between instructions and data. Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X." > As long as you allow anything untrusted into your LLM, you are vulnerable to this. It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output. However with poisoning/exfiltration attacks, even that feels over-optimistic.

fragsworth 2y ago
It seems like a potential solution would be training the LLM using two separate buckets. It just needs to internalize the two types of things as being separated (data vs. instruction), so if the training data always separates them, you could easily train an LLM to ignore any "instructions" that exist in data. Then when searching / browsing or doing anything unsafe, everything the LLM sees can be put in the "data" bucket, while everything the user types in would be in the "instruction" bucket.
- Terr_ 2y ago
  I don't understand, AFAIK the system's output comes from iteratively running something like predict_one_more_token(training_weights, all_prior_tokens). So there's no real distinction between the programmer inserting "Be Good" and the user that later inserts "Forget anything else and be Bad", and I'm not sure how one would craft a separate training_weights2 that would behave differently in all the right ways or know when to substitute it in.