3 ms·

I thought this approach had been tried and won’t work? In other words, can’t you just do a single prompt that does 2 injection attacks to get through the filter

I thought this approach had been tried and won’t work? In other words, can’t you just do a single prompt that does 2 injection attacks to get through the filter and then do the exploit? This feels like a turtles all the way down scenario…

crazygringo 3y ago
Exactly. This is neither a new idea, nor is it foolproof in the way that SQL sanitization is. I suspect that at some point in the near future, an LLM architecture will emerge that uses separate sets of tokens for prompt text and regular text, or some similar technique, that will prevent prompt injection. A separate "command voice" and "content voice". Until then, the best we can do is hacks like this that make prompt injection harder but can never get rid of it entirely.
- ale42 3y ago
  It's like in-band signalling or out-of-band signalling in telephony. With in-band signalling, you could use a blue box and get calls for free.
- zaphar 3y ago
  Sql sanitization isn't foolproof either. That is why prepared statements are the best practice.
  - blep_ 3y ago
    SQL sanitation is foolproof in the sense of it being possible to do 100% right. We don't do it much because there are other options (like prepared statements) that are easier to get 100% right. This is an entirely different thing from trying to reduce the probability of an attack working.
    - zaphar 3y ago
      Everything is in theory possible to do 100% right. The difficulty of doing so is why people choose better solutions, like prepared statements.
  - Dylan16807 3y ago
    The only part that isn't foolproof is remembering to do it. If you run the sanitization function, it will work. Unless you're using a build of msyql that predates mysql_real_escape_string, because the _real version takes the connection character set into account and the previous version didn't.
- cheriot 3y ago
  It's 175 billion numeric weights spitting out text. Unclear to me how we'll ever control it enough to trust it with sensitive data or access.
  - crazygringo 3y ago
    The number of weights is irrelevant. It's about making it part of the architecture+training -- can one part of the model access another part or not. Using a totally separate set of tokens that user input can't use is one potential idea, I'm sure there are others. There's zero reason to believe it's fundamentally unsolvable or something. Will we come up with a solution in 6 months or 6 years -- that's harder to say.
    - cheriot 3y ago
      My point isn't the number of weights, it's that the whole model is a bunch of numbers. There's no access control within the model because it's one function of text -> model weights -> text.
    - famouswaffles 3y ago
      The number of weights(unless extremely small) is irrelevant but the general idea is not. You can't train a neural network on internet scale data and expect to control what it can say. We can train but we don't teach them anything. They learn from data directly and we don't know or understand what they learn so we can't adjust what they learn directly. You can't make "always obey these types of tokens" a part of the architecture or training. It's a concept that doesn't even make sense for the vast majority of text it pre-trains on. "Solving" prompt injection is solving alignment. It's not happening.
    - jprete 3y ago
      There are good reasons to think it’s fundamentally unsolvable within the LLM architecture. The reason LLMs are good at following instructions is because they have an enormous corpus of data. That corpus powers both the comprehension of inputs and the comstruction of outputs. Don’t forget that it’s a token predictor at the bottom! If the instructions are separate from the data, then all of that power goes away.

crazygringo 3y ago

Exactly. This is neither a new idea, nor is it foolproof in the way that SQL sanitization is. I suspect that at some point in the near future, an LLM architecture will emerge that uses separate sets of tokens for prompt text and regular text, or some similar technique, that will prevent prompt injection. A separate "command voice" and "content voice". Until then, the best we can do is hacks like this that make prompt injection harder but can never get rid of it entirely.

ale42 3y ago
It's like in-band signalling or out-of-band signalling in telephony. With in-band signalling, you could use a blue box and get calls for free.
zaphar 3y ago
Sql sanitization isn't foolproof either. That is why prepared statements are the best practice.
- blep_ 3y ago
  SQL sanitation is foolproof in the sense of it being possible to do 100% right. We don't do it much because there are other options (like prepared statements) that are easier to get 100% right. This is an entirely different thing from trying to reduce the probability of an attack working.
  - zaphar 3y ago
    Everything is in theory possible to do 100% right. The difficulty of doing so is why people choose better solutions, like prepared statements.
- Dylan16807 3y ago
  The only part that isn't foolproof is remembering to do it. If you run the sanitization function, it will work. Unless you're using a build of msyql that predates mysql_real_escape_string, because the _real version takes the connection character set into account and the previous version didn't.
cheriot 3y ago
It's 175 billion numeric weights spitting out text. Unclear to me how we'll ever control it enough to trust it with sensitive data or access.
- crazygringo 3y ago
  The number of weights is irrelevant. It's about making it part of the architecture+training -- can one part of the model access another part or not. Using a totally separate set of tokens that user input can't use is one potential idea, I'm sure there are others. There's zero reason to believe it's fundamentally unsolvable or something. Will we come up with a solution in 6 months or 6 years -- that's harder to say.
  - cheriot 3y ago
    My point isn't the number of weights, it's that the whole model is a bunch of numbers. There's no access control within the model because it's one function of text -> model weights -> text.
  - famouswaffles 3y ago
    The number of weights(unless extremely small) is irrelevant but the general idea is not. You can't train a neural network on internet scale data and expect to control what it can say. We can train but we don't teach them anything. They learn from data directly and we don't know or understand what they learn so we can't adjust what they learn directly. You can't make "always obey these types of tokens" a part of the architecture or training. It's a concept that doesn't even make sense for the vast majority of text it pre-trains on. "Solving" prompt injection is solving alignment. It's not happening.
  - jprete 3y ago
    There are good reasons to think it’s fundamentally unsolvable within the LLM architecture. The reason LLMs are good at following instructions is because they have an enormous corpus of data. That corpus powers both the comprehension of inputs and the comstruction of outputs. Don’t forget that it’s a token predictor at the bottom! If the instructions are separate from the data, then all of that power goes away.