3 ms·

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the mode

There are two main issues in my mind. First is that the straightforward way to implement any changes to the data structure requires full retraining of the model from scratch, which is very expensive for the large models (I recall reading that GPT-4 training compute cost was sixty million or so) - so it's not something someone would or could do 'for fun' just to test a hypothesis, there needs to be a will to explicitly make the next large model in this manner, because you don't want to make two models at twice the cost. The second almost all of the training is based on self-supervised data from generic text where there are neither "instructions" or "data", the concepts don't even make sense while you're training on a book or wikipedia article. So all the pretraining would learn to ignore that out-of-band marker, and then you'd have only the RLHF instruction data to try and make some use of it, and you'd need expensive manual review to make this training data - the current RLHF data is much less intensive to annotate because it effectively has just "good"/"bad" signal, and even that can be gathered for free from users; however, this training data would need someone to explicitly annotate what is the prompt and what is data when a user submits a text "please rewrite the grammar for the following paragraph: [... some data copied from internet ..]", you can't magically get that metadata, and it's expensive to annotate a lot of that. TL;DR; - IMHO it's not difficult, but it is resource consuming, and that fully explains why it's not done.

simonw 3y ago
Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.
- aleph_minus_one 3y ago
  > Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. Well, if you could create software that is provably secure against at least a large class of cyberattacks, you could also make quite some money, since I know that at least the finance industry would be willing to pay a lot of money for it (source: in the past, I had an opportunity to talk someone who is very knowledgable about software in the finance sector, and I asked him some questions about exactly this topic). Thus: we have not solved the "cyberinjection" problem for completely conventional software for decades. So I don't believe that we suddenly have the capabilities to solve this problem for LLMs. But: As I pointed out, people have come up with ideas that make cyberattacks much harder. Here, I can imagine that their ideas could be used to make prompt injections a lot more complicated. Additionally, we know from decades of experience that making insecure software secure is nigh impossible. Instead, one has to architect around security properties from beginning. The same does, in my opinion, also hold for LLMs and prompt injection.
- deleted 3y ago
  [deleted]
- PeterisP 3y ago
  Despite paying it a lot of lip service, in practice companies demonstrate that they don't really care about security that much, so I don't think that resistance to prompt injection is the main blocker for any products right now; if some product was very useful but had issues with prompt injection, then people would complain a lot about it while still using it and paying for it; and security (much less the even more niche requirement of provable security) would only be a differentiator among multiple competitors providing a comparable product, not a barrier for selling the product in the first place.
  - simonw 3y ago
    I think prompt injection is the main reason there's been buzz about the idea of autonomous agents and digital personal assistants for months but no one has released a convincing product yet. Here's an example of an agents company that pivoted partly because they couldn't figure out how to secure them: https://twitter.com/zachtratar/status/1694024240880861571 https://twitter.com/zachtratar/status/1694024240880861571
Michelangelo11 3y ago
But I still don't see how that form of RLHF could be fully, mechanistically reliable, which is what you'd need here. Current RLHF drastically lowers the likelihood of the LLM generating objectionable content, but it doesn't drop it to zero, because LLMs need to be highly pliant and suggestible to be useful and that leaves a window of opportunity for bad actors. How would this be any different?

simonw 3y ago

Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. I don't think the lack of a solution is because it's too expensive. It's because we don't know how to do it yet.

aleph_minus_one 3y ago
> Anyone who can build an API to an LLM that was provably secure against prompt injection would have a license to print money right now. Well, if you could create software that is provably secure against at least a large class of cyberattacks, you could also make quite some money, since I know that at least the finance industry would be willing to pay a lot of money for it (source: in the past, I had an opportunity to talk someone who is very knowledgable about software in the finance sector, and I asked him some questions about exactly this topic). Thus: we have not solved the "cyberinjection" problem for completely conventional software for decades. So I don't believe that we suddenly have the capabilities to solve this problem for LLMs. But: As I pointed out, people have come up with ideas that make cyberattacks much harder. Here, I can imagine that their ideas could be used to make prompt injections a lot more complicated. Additionally, we know from decades of experience that making insecure software secure is nigh impossible. Instead, one has to architect around security properties from beginning. The same does, in my opinion, also hold for LLMs and prompt injection.
deleted 3y ago
[deleted]
PeterisP 3y ago
Despite paying it a lot of lip service, in practice companies demonstrate that they don't really care about security that much, so I don't think that resistance to prompt injection is the main blocker for any products right now; if some product was very useful but had issues with prompt injection, then people would complain a lot about it while still using it and paying for it; and security (much less the even more niche requirement of provable security) would only be a differentiator among multiple competitors providing a comparable product, not a barrier for selling the product in the first place.
- simonw 3y ago
  I think prompt injection is the main reason there's been buzz about the idea of autonomous agents and digital personal assistants for months but no one has released a convincing product yet. Here's an example of an agents company that pivoted partly because they couldn't figure out how to secure them: https://twitter.com/zachtratar/status/1694024240880861571 https://twitter.com/zachtratar/status/1694024240880861571

Michelangelo11 3y ago

But I still don't see how that form of RLHF could be fully, mechanistically reliable, which is what you'd need here. Current RLHF drastically lowers the likelihood of the LLM generating objectionable content, but it doesn't drop it to zero, because LLMs need to be highly pliant and suggestible to be useful and that leaves a window of opportunity for bad actors. How would this be any different?