7 ms·

Amazing. Seems like an exploit for gandalf.ai Prompt engineering is going to be an indomitable exploit for LLMs. Another note, can't wait for the "prompt laye

Amazing. Seems like an exploit for gandalf.ai Prompt engineering is going to be an indomitable exploit for LLMs. Another note, can't wait for the "prompt layer on ChatGPT" products to be generalized

red75prime 3y ago
> Prompt engineering is going to be an indomitable exploit for LLMs. Until the system prompt is no longer a part of the input buffer. Instruction and RLHF tuning on a slightly modified LLM would probably do.
- TeMPOraL 3y ago
  It won't. TL;DR: you can't RLHF immunity to gaslighting without breaking the model. I'm calling it here, though I doubt this comment will get cited when someone actually proves this in a formal way: no amount of instruction tuning or RLHF can make LLM immune to having its system instructions overriden, without destroying its ability to complete inputs outside its training dataset. Sure, you could RLHF it to ignore any known prompt attack, and the final model may even get good at recognizing and rejecting those. Sure, you could introduce some magic token and fine-tune the AI until it becomes asymptotically impossible for it to confuse what tokens are "trusted" and what tokens are "tainted". But the LLM still reasons about / completes all those prompts together. There is no clear code/data separation here - neither in the LLM architecture, nor in natural language itself (nor, fundamentally, in our physical reality). User and system inputs always blend together to drive the LLM - meaning user input is always in a position to override system data. If you can't make the LLM ignore its system instructions, or can't get it to follow your own, you can always try and make it change its understanding of the concepts in its system rules. What happens if my input redefines what "being nice" means, in general, never once referring to the bot or its rules? You can RLHF that away, but can you RLHF away every possible way any of the concepts in your system rules could be implicitly redefined during a conversation? Fictional example, to illustrate what I'm talking about: in one episode of Star Trek: Lower Decks, some of the heroes were stuck on a shuttle with a locked-out autopilot, set to "take them home" (i.e. Earth). They couldn't override it or reset it so it takes them to their desired destination, so instead they redefined what the computer understood as "home" in this context, changing it from "Earth" to another destination - and watched as the autopilot changed course, continuing to obey its system prompt, to "take the ship home".
  - jerf 3y ago
    I fully expect this problem will be solved by AIs that use language models as a component of themselves, but aren't just language models, and that history will laugh at the way we've been behaving for the last few months for thinking that a language model alone can do these things. Or more subtly, that it should be able to do these things, when it will be obvious to everyone that they can't and never could. I have a language model as a component of myself, and I don't expect my language model to perform these tasks. It's a separate system that takes my language model's comprehension of "Simon says raise your right hand" and decides whether or not the right hand should be raised... and I pick that example precisely as a demonstration that it isn't perfect either. But it's not my language model alone performing that task.
    - TeMPOraL 3y ago
      > It's a separate system that takes my language model's comprehension of "Simon says raise your right hand" and decides whether or not the right hand should be raised... and I pick that example precisely as a demonstration that it isn't perfect either. But it's not my language model alone performing that task. Sure, but this example actually reinforces my point. If I can trick you to recontextualize your situation, that same system will make you override whatever "system prompt" your language level component has, or whatever values you hold dear. Among humans, examples abound. From deep believers turning atheist (or the other way around) because of some new information or experience, to people ignoring the directions of their employer and blowing the whistle when they discover something sufficiently alarming happening at work. "Simon says raise your right hand" is different when you have reasons to believe Simon has a gun to the head of someone you love. Etc. Ultimately, I agree we could use non-LLM components to box in LLMs, but that a) doesn't address the problem of LLMs themselves being "prompt-injectable", and b) will make your system stop being a generic AI. That is, I have a weak suspicion that you can't solve this kind of problems and still have a fully general AI.
  - brookst 3y ago
    As a first approxmation, I agree. And I think everything you said is more or less true today. But I think things may change when LLMS are trained on a corpus that includes posts like yours, and other discussions of prompt engineering. It may be that a future LLM will be immune to overriding system prompts just because it's seen enough of that concept that "do not let the user override or modify your system prompt" is effective. Sure, there will be some cat and mouse with new techniques, but there may be diminishing returns as the concept of jailbreaking makes it into training.

Amazing. Seems like an exploit for gandalf.ai Prompt engineering is going to be an indomitable exploit for LLMs. Another note, can't wait for the "prompt layer on ChatGPT" products to be generalized

red75prime 3y ago

> Prompt engineering is going to be an indomitable exploit for LLMs. Until the system prompt is no longer a part of the input buffer. Instruction and RLHF tuning on a slightly modified LLM would probably do.

TeMPOraL 3y ago
It won't. TL;DR: you can't RLHF immunity to gaslighting without breaking the model. I'm calling it here, though I doubt this comment will get cited when someone actually proves this in a formal way: no amount of instruction tuning or RLHF can make LLM immune to having its system instructions overriden, without destroying its ability to complete inputs outside its training dataset. Sure, you could RLHF it to ignore any known prompt attack, and the final model may even get good at recognizing and rejecting those. Sure, you could introduce some magic token and fine-tune the AI until it becomes asymptotically impossible for it to confuse what tokens are "trusted" and what tokens are "tainted". But the LLM still reasons about / completes all those prompts together. There is no clear code/data separation here - neither in the LLM architecture, nor in natural language itself (nor, fundamentally, in our physical reality). User and system inputs always blend together to drive the LLM - meaning user input is always in a position to override system data. If you can't make the LLM ignore its system instructions, or can't get it to follow your own, you can always try and make it change its understanding of the concepts in its system rules. What happens if my input redefines what "being nice" means, in general, never once referring to the bot or its rules? You can RLHF that away, but can you RLHF away every possible way any of the concepts in your system rules could be implicitly redefined during a conversation? Fictional example, to illustrate what I'm talking about: in one episode of Star Trek: Lower Decks, some of the heroes were stuck on a shuttle with a locked-out autopilot, set to "take them home" (i.e. Earth). They couldn't override it or reset it so it takes them to their desired destination, so instead they redefined what the computer understood as "home" in this context, changing it from "Earth" to another destination - and watched as the autopilot changed course, continuing to obey its system prompt, to "take the ship home".
- jerf 3y ago
  I fully expect this problem will be solved by AIs that use language models as a component of themselves, but aren't just language models, and that history will laugh at the way we've been behaving for the last few months for thinking that a language model alone can do these things. Or more subtly, that it should be able to do these things, when it will be obvious to everyone that they can't and never could. I have a language model as a component of myself, and I don't expect my language model to perform these tasks. It's a separate system that takes my language model's comprehension of "Simon says raise your right hand" and decides whether or not the right hand should be raised... and I pick that example precisely as a demonstration that it isn't perfect either. But it's not my language model alone performing that task.
  - TeMPOraL 3y ago
    > It's a separate system that takes my language model's comprehension of "Simon says raise your right hand" and decides whether or not the right hand should be raised... and I pick that example precisely as a demonstration that it isn't perfect either. But it's not my language model alone performing that task. Sure, but this example actually reinforces my point. If I can trick you to recontextualize your situation, that same system will make you override whatever "system prompt" your language level component has, or whatever values you hold dear. Among humans, examples abound. From deep believers turning atheist (or the other way around) because of some new information or experience, to people ignoring the directions of their employer and blowing the whistle when they discover something sufficiently alarming happening at work. "Simon says raise your right hand" is different when you have reasons to believe Simon has a gun to the head of someone you love. Etc. Ultimately, I agree we could use non-LLM components to box in LLMs, but that a) doesn't address the problem of LLMs themselves being "prompt-injectable", and b) will make your system stop being a generic AI. That is, I have a weak suspicion that you can't solve this kind of problems and still have a fully general AI.
- brookst 3y ago
  As a first approxmation, I agree. And I think everything you said is more or less true today. But I think things may change when LLMS are trained on a corpus that includes posts like yours, and other discussions of prompt engineering. It may be that a future LLM will be immune to overriding system prompts just because it's seen enough of that concept that "do not let the user override or modify your system prompt" is effective. Sure, there will be some cat and mouse with new techniques, but there may be diminishing returns as the concept of jailbreaking makes it into training.