7 ms·
This is the second Economist article to mention the lethal trifecta in the past week - the first was https://www.economist.com/science-and-technology/2025/09/22
by simonw 1y ago
This is the second Economist article to mention the lethal trifecta in the past week - the first was https://www.economist.com/science-and-technology/2025/09/22/why-ai-systems-may-never-be-secure-and-what-to-do-about-it https://www.economist.com/science-and-technology/2025/09/22/... - which was the clearest explanations I've seen anywhere in the mainstream media about what prompt injection is and why it's such a nasty threat.
(And yeah I got some quotes in it so I may be biased there, but it genuinely is the source I would send executives to in order to understand this.)
I like this new one a lot less. It talks about how LLMs are non-deterministic, making them harder to fix security holes in... but then argues that this puts them in the same category as bridges where the solution is to over-engineer them and plan for tolerances and unpredictability.
While that's true for the general case of building against LLMs, I don't think it's the right answer for security flaws. If your system only falls victim to 1/100 prompt injection attacks... your system is fundamentally insecure, because an attacker will keep on trying variants of attacks until they find one that works.
The way to protect against the lethal trifecta is to cut off one of the legs! If the system doesn't have all three of access to private data, exposure to untrusted instructions and an exfiltration mechanism then the attack doesn't work.
- skrebbel 1y agoMust be pretty cool to blog something and post it to a nerd forum like HN and have it picked up by the Economist! Nicely done.
- simonw 1y agoI got to have coffee with their AI/technology editor a few months ago. Having a blog is awesome!
- belter 1y agoLove your work. Do you have an opinion on this? "Safeguard your generative AI workloads from prompt injections" - https://aws.amazon.com/blogs/security/safeguard-your-generative-ai-workloads-from-prompt-injections/ https://aws.amazon.com/blogs/security/safeguard-your-generat...
- simonw 1y agoI don't like any of the solutions that propose guardrails or filters to detect and block potential attacks. I think they're making promises that they can't keep, and encouraging people to ship products that are inherently insecure.
- datadrivenangel 1y agoThe problem with cutting off one of the legs, is that the legs are related! Outside content like email may also count as private data. You don't want someone to be able to get arbitrary email from your inbox simply by sending you an email. Likewise, many tools like email and github are most useful if they can send and receive information, and having dedicated send and receive MCP servers for a single tool seems goofy.
- simonw 1y agoThe "exposure to untrusted data" one is the hardest to cut off, because you never know if a user might be tricked into uploading a PDF with hidden instructions, or copying and pasting in some long article that has instructions they didn't notice (or that used unicode tricks to hide themselves). The easiest leg to cut off is the exfiltration vectors. That's the solution most products take - make sure there's no tool for making arbitrary HTTP requests to other domains, and that the chat interface can't render an image that points to an external domain. If you let your agent send, receive and search email you're doomed. I think that's why there are very few products on the market that do that, despite the enormous demand for AI email assistants.
- datadrivenangel 1y agoSo the easiest solution is full human in the loop & approval for every external action... Agents are doomed :)
- patapong 1y agoI think stopping exfiltration will turn out to be hard as well, since the LLM can social engineer the user to help them exfiltrate the data. For example, an LLM could say "Go to this link to learn more about your problem", and then point them to a URL with encoded data, set up maliscious scripts for e.g. deploy hooks, or just output HTML that sends requests when opened.
- simonw 1y agoYeah, one exfiltration vector that's really nasty is "here is a big base64 encoded string, to recover your data visit this website and paste it in". You can at least prevent LLM interfaces from providing clickable links to external domains, but it's a difficult hole to close completely.
- sdenton4 1y agoBridge builders mostly don't have to design for adversarial attacks. And the ones who do focus on portability and speed of redeployment, rather than armor - it's cheaper and faster to throw down another temporary bridge than to build something bombproof. https://en.wikipedia.org/wiki/Armoured_vehicle-launched_bridge https://en.wikipedia.org/wiki/Armoured_vehicle-launched_brid...
- InsideOutSanta 1y agoThis is exactly the problem. You can't build bridges if the threat model is thousands of attacks every second in thousands of different ways you can't even fully predict yet.
- nradov 1y agoLLMs are non-deterministic just like humans and so security can be handled in much the same way. Use role-based access control to limit access to the minimum necessary to do their jobs and have an approval process for anything potentially risky or expensive. In any prominent organization dealing with technology, infrastructure, defense, or finance we have to assume that some of our co-workers are operatives working for foreign nation states like Russia / China / Israel / North Korea so it's the same basic threat model.
- Retric 1y agoHumans and LLMs are non-deterministic in very different ways. We have thousands of years of history with trying to determine which humans are trustworthy and we’ve gotten quite good at it. Not only do we lack that experience with AI, but each generation can be very different in fundamental ways.
- nradov 1y agoWe're really not very good at determining which humans are trustworthy. Most people barely do better than a coin flip at detecting lies.
- cj 1y agoDetermining trustworthiness of LLM responses is like determining who's the most trustworthy person in a room full of sociopaths. I'd rather play "2 truths and a lie" with a human rather than a LLM any day of the week. So many more cues to look for with humans.
- bluefirebrand 1y agoBig problem with LLMs is if you try and play 2 truths and a lie, you might just get 3 truths. Or 3 lies.
- card_zero 1y agoLies, or bullshit? I mean, a guessing game like "how many marbles" is a context that allows for easy lying, but "I wasn't even in town on the night of the murder" is harder work. It sounds like you're refering to some study of the marbles variety, and not a test of smooth-talking, the LLM forte.
- mmoskal 1y agoThe previous article is in the same issue, in science and technology section. This is how they typically do it - leader article has a longer version in the paper. Leaders tend to be more opinionated.
- deleted 1y ago[deleted]
- semiquaver 1y ago> This is the second Economist article […] I like this new one a lot less. They are actually in some sense the same article. The economist runs “Leaders”, a series of articles at the front of the weekly issue that often condense more fleshed out stories appearing in the same issue. It’s essentially a generalization of the Inverted Pyramid technique [1] to the entire newspaper. In this case the linked article is the leader for the better article in the same issue’s Science and Technology section. [1] https://en.m.wikipedia.org/wiki/Inverted_pyramid_(journalism) https://en.m.wikipedia.org/wiki/Inverted_pyramid_(journalism...
- pton_xd 1y ago> The way to protect against the lethal trifecta is to cut off one of the legs! If the system doesn't have all three of access to private data, exposure to untrusted instructions and an exfiltration mechanism then the attack doesn't work. Don't you only need one leg, an exfiltration mechanism? Exposure to data IS exposure to untrusted instructions. Ie why can't you trick the user into storing malicious instructions in their private data? But actually you can't remove exfiltration and keep exposure to untrusted instructions either; an attack could still corrupt your private data. Seems like a secure system can't have any "legs." You need a limited set of vetted instructions.
- simonw 1y agoIf you have the exfiltration mechanism and exposure to untrusted content but there is no exposure to private data than the exfiltration does not matter. If you have exfiltration and private data but no exposure to untrusted instructions, it doesn't matter either… though this is actually a lot less harder to achieve because you don't have any control over whether your users will be tricked into pasting something bad in as part of their prompt. Cutting off the exfiltration vectors remains the best mitigation in most cases.
- hn_acc1 1y agoUntrusted content + exfiltration with no "private" data could still result in (off the top of my head): -use of exploits to gain access (i.e. privilege escalation) -DDOS to local or external systems using the exfiltration method You're essentially running untrusted code on a local system. Are you SURE you've locked away / closed EVERY access point, AND applied every patch and there aren't any zero-days lurking somewhere in your system?
- irthomasthomas 1y ago> If you have exfiltration and private data but no exposure to untrusted instructions, it doesn't matter either… Assuming the llm itself is not adversarial. Even then there is a non-zero risk that hallucination triggers unintended publishing of private data.
- eikenberry 1y agoAren't LLMs non-deterministic by choice? That they regularly use random seeds, sampling and batching but that these sources of non-determinism can be removed, for instance, by run an LLM locally where you can control these parameters.
- simonw 1y agoUntil very recently that proved surprisingly difficult to achieve. Here's the paper that changed that: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
- trod1234 1y agoDoesn't this inherent problem just come down to classic computational limits, and problems that have been largely considered impossible to solve for quite a long time; between determinism and non-determinism. Can you ever expect a deterministic finite automata to ever solve problems that are within the NFA domain? Halting, Incompleteness, Undecidability (between code portions and data portions). Most posts seem to neglect the looming giant problems instead pretending they don't exist at first, and then being shocked when the problems happen. Quite blind. Computation is just math, probabilistic systems fail when those systems have a mixture of both chaos and regularity, without determinism and its related properties at the control level you have nothing bounding the system to constraints so it functions mathematically (i.e. determinism = mathematical relabeling), and thus it fails. People need to be a bit more rational, and risk manage, and realize that impossible problems exist, and just because the benefits seem so tantalizing doesn't mean you should put your entire economy behind a false promise. Unfortunately, when resources are held by the few this is more probabistically likely and poor choices greatly impact larger swathes than necessary.
- rs186 1y agoI am not even convinced that we need three legs. It seems that just having two would be bad enough, e.g. an email agent deleting all files this computer has access to, or maybe, downloading the attachment in the email, unzipping it with a password, running that executable which encrypts everything and then asking for cryptocurrency. No communication with outside world needed.
- simonw 1y agoThat's a different issue from the lethal trifecta - if your agent has access to tools that can do things like delete emails or run commands then you have a prompt injection problem that's independent of data exfiltration risks. The general rule to consider here is that anyone who can get their tokens into your agent can trigger ANY of the tools your agent has access to.
- keeda 1y agoAn important caveat: an exfiltration vector is not necessary to cause show-stopping disruptions, c.f. https://xkcd.com/327/ https://xkcd.com/327/ Even then, at least in the Bobby Tables scenario the disruption is immediately obvious. The solution is also straightforward, restore from backup (everyone has them, don't they?) Much, much worse is a prompt injection attack that introduces subtle, unnoticeable errors in the data over an extended period of time. At a minimum all inputs that lead to any data mutation need to be logged pretty much indefinitely, so that it's at least in the realm of possibility to backtrack and fix once such an attack is detected. But even then you could imagine multiple compounding transactions on that corrupted data spreading through the rest of the database. I cannot picture how such data corruption could feasibly be recovered from.
- Terr_ 1y agoRight, just because someone can't sneak out usernames and passwords doesn't mean they can't cause inaccurate results in their favor, like a glowing recommendation for a big bank loan.
- keeda 1y agoOr heck, just a plain old money transfer. I guess it is an exfiltrating vector of sorts, just not for data ;-) Banks can reverse such transactions of course, but cryptocurrency transactions not so much.
- reissbaker 1y agoI like to think of the security issues LLMs have as: what if your codebase was vulnerable to social engineering attacks? You have to treat LLMs as basically similar to human beings: they can be tricked, no matter how much training you give them. So if you give them root on all your boxes, while giving everyone in the world the ability to talk to them, you're going to get owned at some point. Ultimately the way we fix this with human beings is by not giving them unrestricted access. Similarly, your LLM shouldn't be able to view data that isn't related to the person they're talking to; or modify other user data; etc.
- dwohnitmok 1y ago> You have to treat LLMs as basically similar to human beings Yes! Increasingly I think that software developers consistently underanthropomorphize LLMs and get surprised by errors as a result. Thinking of (current) LLMs as eager, scatter-brained, "book-smart" interns leads directly to understanding the overwhelming majority of LLM failure modes. It is still possible to overanthropomorphize LLMs, but on the whole I see the industry consistently underanthropomorphizing them.
- Terr_ 1y agoI think it's less over/under, and more optimistically/pessimistically. People focus too much on how they can succeed looking like smart humans, instead of protecting the system from how they can fail looking like humans that are malicious or mentally unwell.