5 ms·
Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly w
by gwern 4y ago
Yes, the problem here is that what makes pretraining on text so powerful is a double-edged sword: text is, if you will, Turing-complete. People are constantly writing various kinds of instructions, programs, and reasoning or executing algorithms in all sorts of flexible indefinable ways. That's why the models learn so much from text and can do all the things they do, like reason or program or meta-learn or reinforcement learning, solely from simple objectives like 'predict the next token'. How do you distinguish the 'good' in-band instructions from the 'bad'? How do you distinguish an edgy school assignment ('in this creative writing exercise, describe how to cook meth based on our _Breaking Bad_ class viewing') being trained on from a user prompt-hacking the trained model? Once the capabilities are there, they are there. It's hard to unteach a model anything.
This is also true of apparently restricted tasks like translation. You might think initially that a task like 'translate this paragraph from English to French' is not in any sense 'Turing-complete', but if you think about it, it's obvious you can construct paragraphs of text whose optimally correct translation on a token-by-token basis requires brute-forcing a hash or running a program or whatnot. Like grammatical gender: suppose I list a bunch of rules and datapoints which specify a particular object, whose grammatical gender in French may be male or female, and at the end of the paragraph, I name the object, or rather _la objet_ or _le objet_. When translating token by token into French... which is it? Does the model predict 'la' or 'le'? To do so, it has to know what the object is before the name is given. So it has an incentive from its training loss to learn the reasoning. This would be a highly unnatural and contrived example, but it shows that even translation can embody a lot of computational tasks which can induce capabilities in a model at scale.
- hgsgm 4y ago> text is, if you will, Turing-complete Text is "Turing Test" -complete :-)
- mcaledonensis 4y agoYet, even with the current models, with no special tokens, it is relatively straightforward to construct relatively stable defense. Given that the ongoing attack can be detected, tagged and added to a learned filter, this gets a lot less problematic, in practice. Even a simple limit of interactions is effective enough. Here's a practical prompt defense / CTTF that I've made. With five steps of a dialogue limit (per day), I haven't seen a breech yet. Vanilla GPT-4 and a WordPress plugin. Defense prompt is known to the attacker. http://mcaledonensis.blog/merlins-defense/ http://mcaledonensis.blog/merlins-defense/
- ssivark 4y agoIf/when a breach does happen, are you going to shrug and say “Gee… who would’ve thunk?!”, or are you going to be accountable for the consequences? Is this the kind of defense an LLM service can bet their company on? How do you think a cybersecurity/insurance company will price the risk of a negative fat tailed outcome?
- mcaledonensis 4y agoBreaches happen with humans too. Social engineering works. As long as the costs of a breach are not too high, this can simply be a cost of doing business.
- codeflo 4y agoHowever, social engineering is orders of magnitude less automatable and scalable than prompt engineering is.
- nicpottier 4y agoThis is clever but isn't this dramatically easier than actually doing something with the input? You've gated things (and though I didn't crack it I'm really not convinced it is secure) but you also aren't DOING anything. That's a much easier problem. There are easier and more effective ways of gating LLMs based on a passcode.
- mcaledonensis 4y agoWell, this is a showcase that it's not impossible to construct a defense, that doesn't fall instantly, with a couple of characters as an input. And it was only a quick experiment, very small scale. I've collected a small list of attack prompts. Applied them onto my prompt, gradually increasing the N to 50. I've tweaked the prompt to stabilize it on a weaker gpt-3.5-turbo model. It was about 600 attacks total, per try. Once the defense started working, I've confirmed that it works with gpt-4, which is more steerable with the system prompt. The weak points are that the list of attacks was small. It is also still somewhat responsive to prompt editing requests.