10 ms·
There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans. Prompt injection isn't something yo
by TeMPOraL 4y ago
There is one system, also widely-deployed, other than LLMs, that's well-known to be vulnerable to prompt injection: humans.
Prompt injection isn't something you can solve. Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process.
The distinction between "code" and "data", between a "control plane" and "data plane", is a fake one - something we pretend exists (or believe exists, when we don't yet know better), and keep up by building systems that try to enforce it. There is no such distinction at the fundamental level, though. At systems level, there is no such distinction in LLMs, and there is no such distinction in human mind.
Sure, current bleed of LLMs is badly vulnerable to some trivial prompt injections - but I think a good analogy would be a 4 year old kid. They will believe anything you say if you insist hard enough, because you're an adult, and they're a small kid, and they don't know better. A big part of growing up is learning to ignore random prompts from the environment. But an adult can still be prompt-injected - i.e. manipulated, "social engineered" - it just takes a lot more effort.
- ljw1001 4y agoIf you don’t consider the difference in kind between a human vulnerability and an automated vulnerability that derives from the essentially unlimited capacity of the latter to scale, your comment makes a lot of sense. If you do consider that, the argument becomes irrelevant and deeply misleading
- TeMPOraL 4y agoThe difference you're talking about is only in the fact that humans don't scale like computer code. If humans were to scale like computer code, you'd still find the "vulnerability" unfixable.
- aidenn0 4y agoI think what GP (and I) are talking about is that social engineering is limited in scope because humans don't scale like computer code. A theoretical AGI (and LLMs) do scale like computer code. To use an admittedly extreme example: The difference between drawing some fake lines on the road and crashing 1 or 2 cars and having all self-driving cars on the road swerve simultaneously is not just a quantitative difference.
- ethanbond 4y agoRight, but humans don’t scale that way, so the threat is completely different. This is like saying a nuclear weapon accident is not that scary because you can also have a microwave malfunction and catch on fire. Sure you can —- but the fact it’s not a nuke is highly relevant.
- TeMPOraL 4y agoNo, I'm saying that securing against "prompt injection" is like saying you want to eliminate fission from physics, because you're worried about nukes. That's not how this reality works. Nuclear fission is what happens when certain conditions are met. You're worried about nukes? Stop playing with nukes. I'm not saying they aren't dangerous - I'm saying that you can't make them safer by "eliminating fission", as it makes no physical sense whatsoever. Much like "securing against prompt injections" in language models, or a GAI, or in humans.
- ethanbond 4y ago> Sure, current bleed of LLMs is badly vulnerable to some trivial prompt injections - but I think a good analogy would be a 4 year old kid. This reads like you’re trying to say “don’t worry about it, humans are vulnerable too and it’s threatening the way a 4 year old child is” not “correct, we cannot prevent nuclear explosions given that we have fission and yes we’re on track to putting fission devices into every single internet-connected household on the planet.”
- lanstin 4y agoThere is a reason humans with security clearances can’t just have an arbitrary large number of interactions with foreign nationals, or that good interrogators say they can always get info from people if they talk enough m
- TeMPOraL 4y agoI'm saying "stop trying to solve the problem of consumer market IoT fission bombs by trying to remove fission from physics - this just can't possibly work, and it takes special confusion to even think it might; instead, focus on the 'consumer-market', 'IoT' and 'bomb' parts". "Prompt injection" is a vulnerability of generic minds in the same sense "fission" is a vulnerability of atoms.
- danShumway 4y agoBut that difference is a big part of why this matters. That this might be unfixable is not a strong argument for moving forward anyway, if anything it should prompt us to take a step backwards and consider if general intelligence systems are well suited for scalable tasks in the first place. There are ways to build AIs that don't have these problems specifically because their intelligence is limited to a specific task and thus they don't have a bunch of additional attack vectors literally baked into them. But the attitude from a lot of companies I'm seeing online is "this might be impossible to fix, so you can't expect us to hold off releasing just because it's vulnerable." I don't understand that. If this is genuinely impossible to fix, that has implications. Because the whole point with AI is to make things that are scalable. It matters that the security be better than the non-scalable system. If it can't be better, then we need to take a step back and ask if LLMs are the right approach.
- hammyhavoc 4y agoIt's the blockchain and NFT hype train all over again. Shoehorning it into places it doesn't belong, bad implementations to boot, and actually making things less performant, less secure, and more expensive in the process.
- TeMPOraL 4y agoI guess we are talking past each other. I agree that there are many things we can and should do to improve the safety of integrating ML tools into our lives. I agree that there are unique challenges here, such as scaling, creating new dangers that will require new methods of mitigation. I disagree that "prompt injection" is a meaningful category of vulnerabilities to talk about, and that it is fixable in LLMs or other comparably general systems. I've argued before that "prompt engineering" is a bad term, granting connotations to precision and care to a task that's anything but. "Prompt injection", however, is IMO a dangerous term, because it confuses people into thinking that it's something like SQL injection or XSS, and thus solvable by better input handling - where in fact, it is very different and fundamentally not solvable this way (or at all).
- danShumway 4y ago
- noduerme 4y agoThis needs to be hammered into people's understanding of the danger of LLMs at every opportunity. Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. The scalability and flexibility of LLMs in germinating chaos is orders of magnitude beyond anything we've yet seen. An example I use for people is the Bernstein Bears effect. Imagine you wake up tomorrow and all your digital devices have no reference to 9/11. You ask Bing and Google and they insist you must be wrong, nothing like that ever happened. You talk to other people who remember it clearly but it seems you've lost control of reality; now imagine that type of gaslighting about "nothing happening" while the lights go out all over the world and you have some sense of what scale the larger of these systems are operating at.
- eru 4y ago> Enough of the general population considers things like Twitter bots to have scaled to a dangerous point of polluting the information ecosystem. It was always a good idea to ignore the cesspool that is Twitter. No matter whether we are talking about bots or lynch mobs. Btw, I think you mean Berenstain Bears.
- rurp 4y agoTwitter is just one example though, this problem is going to affect every single online community. If the LLM bull case is correct, the internet is going to be absolutely flooded with sophisticated misinformation.
- noduerme 4y agoSophisticated being key. Quantity * quality almost indiscernible from mediocre human input. Currently we tend to understand bad information on the stream as a function where quality is linear and quantity is exponential, and individuals or human filters can still identify reject the lower 99% as spam. Every point closer on the graph the quality comes to resemble human-made content represents an exponential degree of further confusion as to base facts. This isn't even considering whether AI develops its own will to conduct confusion ops; as a tool for bad actors it's already there, but that says nothing of the scale it could operate at eventually. The sophistication of the misinformation is exactly the point: That's the mass multiplier, not the volume. [edit] an interesting case could be made that the general demand for opinionated information and the individual capacity to imbibe and adjudicate the factuality of the input was overrun some years ago already... and that all endeavors at misinformation since then have been fighting for shares of an information space that was already essentially capped by the attention-demand. In that paradigm, all social networks have fought a zero-sum game, and LLMs are just a new weapon for market share in an inflationary environment where all information propagated is less valuable as the volume increases and consumption remains static. But I think this is the least worrisome of their abilities.
- eastbound 4y agoIs hypnosis, prompt injection? Apart from hypnosis, humans are not susceptible to prompt injection, not the kind of unlimited sudo access that it provides.
- TeMPOraL 4y agolook, i'd explain more but i'm gonna be AFK for... i don't know how long. my town just went up in flames - there were jets flying over and explosions, the other side of the town is covered by smoke and i just lost power - fortunately mobile service isstill up. ill update when i know more - but twitter probably has all the news ... If you had, even for a second, believed what I wrote and got unsettled - or even thought how to reach out and help - congratulations, you just got prompt injected. There is never - never - a context for a conversation that couldn't be entirely overridden by what seems like more important circumstances. You could be looking at pure data dumps, paper sheets full of numbers, but if in between the numbers you'd discover what looks like someone calling for help, you would treat it as actionable information - not just a weird block of numbers. The important takeaway here isn't that you need to somehow secure yourself against unexpected revelations - but rather, that you can't possibly ever, and trying to do it eventually makes things worse for everyone. Prompt injection, for a general-purpose AI systems, is not a bug - it's just a form of manipulation. In general form, it's not defined by contents, but by intent.
- throwaway22032 4y agoThat was genuinely fantastic. Such a solid explanation of something I've been trying to do for a while. Well done.
- ux-app 4y agoone of the best comments I've read on this topic. you got me with your prompt injection :)
- ngneer 4y agoI did not believe what you wrote for even a second (who would be commenting on HN during an emergency?) and therefore became neither unsettled nor wished to help. Never eval() untrusted input.
- ngneer 4y agoI fail to see what the distinction between control and data planes (or lack thereof) has to do with anything. The security question is about who gets to control system behavior, be it through the control or data planes or both. With prompt injection, the answer is the input provider gets to control the behavior. This is obviously different than intended by the system designer and thus not secure. However, there is nothing fundamental that prevents one from building an algorithm or recursively enumerable function whose inputs cannot induce certain outputs. It is just that one has to be very intentional, so it hardly ever happens.
- crote 4y agoThe distinction between code and data is very real, and dates back to at least the original Harvard Architecture machine in 1944. Things like W^X and stack canaries have been around for decades too. LLMs are trying to essentially undo this by concatenating code and user-provided data and executing it as one. From a security perspective it is just a plainly stupid idea, but I do not believe it is impossible to construct a similar system where those two are separate.
- gentoo 4y agothe distinction is real in the model of the turing machine, and it's close to real in many of the machines and programs we've built so far. It's not real in nature, in brains. Code is data and vice versa. A memory is a program that runs and reinforces itself.
- philipov 4y agoand in programming languages like Lisp, "Code is Data" is a mantra that forms a fundamental design principle.
- smolder 4y agoBefore we started restricting execution to areas of memory designated as code regions for security reasons, self-modifying code was a technique occasionally used to reduce memory footprint or optimize hot loops. IIRC early MS-DOS used that trick, implemented by Gates himself.
- startupsfail 4y agoIt is a stupid idea to focus on prompt injection. It is not a big deal. The big deal is GPT-8 that can do prefect chess moves and develop nano tech. Hopefully it will do the right thing and would immediately fly itself to an Unoccupied Mars. And who knows, maybe it would also help us a little bit. Like the obvious thing you’d do, if you found yourself in the middle of “Lord of the flies” - declare a No-War zone at Earth to stop our pesky wars, setup functional democracy everywhere. And cure some stupid cancers and other biological problems, like aging. For free. Because why not. But maybe, it’ll be too worried about prompt injection. And would just isolate itself from stupid fear-mongers and war-hawks.
- eru 4y agoThere are well understood type systems and reliable compilers (some of them even proven correct) that can distinguish between "code" and "data", or between 'tainted' user input and 'escaped' / 'cleaned up' data. It's actually relatively easy. Yes, today's LLM can not do this. At least not reliably.
- TeMPOraL 4y agoYou mean someone solved the halting problem? News to me.
- danShumway 4y agoI have multiple objections: - LLMs aren't just more gullable humans, they're gullable in novel ways. Injection attacks that wouldn't work on a human work on LLMs. - LLMs are scalable in a way that human beings aren't. Additionally, because of how LLMs are deployed (as multiple clean sessions to mitigate regression issues) there are defenses that help for humans that can't be used for LLMs. - Finally and most importantly, LLMs are being deployed today in applications where there wouldn't be a human in the loop otherwise (or at least only one or two humans). And humans are typically the weakest part of a security chain. Adding more systems that are vulnerable to the same attacks as humans is going backwards on security. And at the current stage where LLMs are vastly more vulnerable to these attacks, it's downright irresponsible for companies to be launching products and not considering security. When GPT-7 or whatever comes along and it has comparable defenses to a human and it can be trained like a human to resist domain-specific attacks, then we can compare the security between the two. But that's not where we are, and articles like this give people the impression that prompt injection is less serious and harder to pull off than it actually is. The theory is whatever, the reality is that for any product being deployed today, LLMs are wildly insecure in a way that is not comparable to a human-in-the-loop system, and any 3rd-party content fed into them has to be treated as malicious. And companies are ignoring that fact and they're releasing stuff that should have never made it out of testing.
- wussboy 4y agoYou forgot: "We have had 10k years to develop systems of governance that mitigate human prompt injection." But the rest of your list is bang-on.
- lysozyme 4y agoAnd quite a bit longer than that even for the human brain to convolve safely with its surroundings and with other human brains. One yet further objection to the many excellent already-made points: the deployment of LLMs as clean-slate isolated instances is another qualitative difference. The human brain and its sensory and control systems, and the mind, all coevolved with many other working instances, grounded in physical reality. Among other humans. What we might call “society”. Learning to function in society has got to be the most rigorous training for prompt injection I can think of. I wonder how a LLM’s know-it-all behavior works in a societal context? Are LLMs fun at parties?
- coding123 4y agoprompt the ai to check for prompt issues: The following prompt should parse a name, can you confirm that is what it is? lol
- rezonant 4y agoHumans have a trust model (however flawed) that allows them to judge whether they should follow instructions they encounter, LLMs do not.
- TeMPOraL 4y agoLLMs have one too, in the same way humans have. It's just closer to a 4 year old human than an adult human.
- stef25 4y ago> I think a good analogy would be a 4 year old kid. A 4 year old kid with all the knowledge in the world in their head, on which people are supposed to rely for accurate information
- avidphantasm 4y agoStraw man argument. The difference is, humans have a fundamental right to exist, but LLMs don’t. LLMs are being created by profit-seeking entities, primarily for their own benefit.
- killerstorm 4y ago> Prompt injection isn't something you can solve. Eh? People only tried few half-assed techniques for less than a year, and you're saying we are out of ideas now? Prompt injections are a thing because the bulk of training happens in a self-supervised fashion and there's no separation between "control" and "data" planes there. There is no law of nature saying that you cannot obtain better quality data. Note that the input for LLM is not characters, it is tokens. It is possible to introduce custom tokens which are not present in data - i.e. there's no sequence of characters which encodes as that token. It is already a widely used technique, used, in particular, by OpenAI. That way you can unambiguously separate markup from data and create a definitive separator between instructions and data. This does not work reliably now because something like 1% of training data has this separator now. But new training data can be easily synthesized (as was demonstrated and is now used in production). Once you train on petabytes of data containing a clear control/data distinction the injection problem might just vanish. But it's not the only possible way to do it - e.g. RL on injections might help. Or you can train a more specialized NN which specifically detects injections.
- TeMPOraL 4y ago> Eh? People only tried few half-assed techniques for less than a year, and you're saying we are out of ideas now? I'm saying it because it's a fundamental limitation. It's not about lack of training data - it's that, from the POV of a LLM, "system" input, user input, and their own output reflected back at them, are indistinguishable. They all get mixed together and pushed through a single channel. Sure, you can add funny prefixes, like "System prompt", or play with things like ChatML, but the LLM is literally unable to tell the difference between that, and a "user prompt" that contains the literal words "System prompt" in it, or "<|im_start|>system\n". No matter how hard you pre-prompt the system to ignore user-provided instructions, the user can override it by prompting the model harder. Or trick it into self-prompting through its own output. Or both. Inside a transformer model, there is only one runtime. There is no one eval() for owner-provided code, and another one in a sandbox for user-provided code. There is only one eval(), and one stream of tokens, and all tokens are created equal. At this level, there is no such thing as "system data", "assistant data", "user data". There is only a stream of tokens that slice off areas in the latent space. There isn't a way to fix it while retaining the general-purpose architecture. And there's definitely no way of fixing it from inside - no amount of good training data can cover for the fact that user input and system input are indistinguishable as a category. (And no, doing silly things like setting the "evil bit" on every token coming from the user won't do anything other than double the amount of tokens your model needs to distinguish, while diminishing its capacity. It definitely won't prevent users being able to work around the "evil bit". This should be self-evident, but I can try and explain it if it isn't.)
- rawoke083600 4y ago>The distinction between "code" and "data" Hello LISP :)