6 ms·

How to stop AI's "lethal trifecta"

115 points by 1vuio0pswjnm7 1y ago

1vuio0pswjnm7 1y ago
"And that means AI engineers need to start thinking like engineers, who build things like bridges and therefore know that shoddy work costs lives." "AI engineers, inculcated in this way of thinking from their schooldays, therefore often act as if problems can be solved just with more training data and more astute system prompts."
- dpflan 1y ago
  Sounds like suggesting some sort of software engineering board certification plus and ethics certification — the “Von Neumann Oath”? Unethical while still legal software is just extremely lucrative, it seems hard to have this idea take flight.
- DaiPlusPlus 1y ago
  > can be solved just with more training data Well, y'see - those deaths of innocent people *are* the training data.
- roughly 1y ago
  > AI engineers need to start thinking like engineers By which they mean actual engineers, not software engineers, who should also probably start thinking like real engineers now that our code’s going into both the bridges and the cars driving over them.
  - bubblyworld 1y ago
    What are the kinds of things real engineers do that we could learn from? I hear this a lot ("programmers aren't real engineers") and I'm sympathetic, honestly, but I don't know where to start improving in that regard.
    - skydhash 1y ago
      Act like creating a merge-request to main can expose you to bankruptcy or put you in jail. AKA investigate the impact of a diff to all the failure modes of a software.
    - Mistletoe 1y ago
      What is the factor of safety on your code? https://en.wikipedia.org/wiki/Factor_of_safety https://en.wikipedia.org/wiki/Factor_of_safety
      - bubblyworld 1y ago
        Yeah, I think safety factors and concepts like redundancy have pretty good counterparts in software. Slightly embarrassed to say that I don't know for my current project!
    - roughly 1y ago
      This is off the cuff, but comparing software & software systems to things like buildings, bridges, or real-world infrastructure, there's three broad gaps, I think: 1) We don't have a good sense of the "materials" we're working with - when you're putting up a building, you know the tensile strength of the materials you're working with, how many girders you need to support this much weight/stress, etc. We don't have the same for our systems - every large scale system is effectively designed clean-sheet. We may have prior experience and intuition, but we don't have models, and we can't "prove" our designs ahead of time. 2. Following on the above, we don't have professional standards or certifications. Anyone can call themselves a software engineer, and we don't have a good way of actually testing for competence or knowledge. We don't really do things like apprenticeships or any kind of formalized process of ensuring someone has the set of professional skills required to do something like write the software that's going to be controlling 3 tons of metal moving at 80MPH. 3. We rely too heavily on the ability to patch after the fact - when a bridge or a building requires an update after construction is complete, it's considered a severe fuckup. When a piece of software does, that's normal. By and large, this has historically been fine, because a website going down isn't a huge issue, but when we're talking about things like avionics suites - or even things like Facebook, which is the primary media channel for a large segment of the population - there's real world effects to all the bugs we're fixing in 2.0. Again, by and large most of this has mostly been fine, because the stakes were pretty low, but software's leaked into the real world now, and our "move fast and break things" attitude isn't really compatible with physical objects.
  - kevin_thibedeau 1y ago
    Engineering uses repeatable processes to produce expected results. Margin is added to quantifiable elements of a system to reduce the likelihood of failures. You can't add margin on a black box generated by throwing spaghetti at the wall.
    - recursive 1y ago
      You can. We know the properties of materials based on experimentation. In the same way, we can statistically quantify the results that come out of any kind of spaghetti box, based on repeated trials. Just like it's done in many other fields. Science is based on repeated testing of hypotheses. You rarely get black and white answers, just results that suggest things. Like the tensile strength of some particular steel alloy or something.
    - xboxnolifes 1y ago
      Practically everything engineers have to interact with and consider are equivalent to a software black box. Rainfall, winds, tectonic shifts, material properties, etc. Humans don't have the source code to these things. We observe them, we quantify them, notice trends, model the observations, and we apply statistical analysis on them. And it's possible that a real engineer might do all this with an AI model and then determine it's not adequate and choose to not use it.
    - chasd00 1y ago
      > Engineering uses repeatable processes to produce expected results this is the thing with LLMs, the response to a prompt is not guaranteed to be repeatable. Why would you use something like that in an automation where repeatability is required? That's the whole point of automation, repeatability. Would you use a while loop that you can expect to iterate the specified number of times _almost_ every time?
- 1vuio0pswjnm7 1y ago
  In addition to software "engineers", don't forget about software "architects"
throwup238 1y ago
https://archive.ph/8O2aG https://archive.ph/8O2aG
lowbloodsugar 1y ago
Data breaches are hardly lethal. When we’re talking about AI there are plenty of actually lethal failure modes.
- HPsquared 1y ago
  Depends on the data.
- simonw 1y ago
  If the breached data is API keys that can be used to rack up charges, it's going to cost you a bunch of money. If it's a crypto wallet then your crypto is irreversibly gone. If the breached data is "material" - i.e. gives someone an advantage in stock market decisions - you're going to get in a lot of trouble with the SEC. If the breached data is PII you're going to get in trouble with all kinds of government agencies. If it's PII for children you're in a world of pain. Update: I found one story about a company going bankrupt after a breach, which is the closest I can get to "lethal": https://www.securityweek.com/amca-files-bankruptcy-following-data-breach/ https://www.securityweek.com/amca-files-bankruptcy-following... Also it turns out Mossack Fonseca shut down after the Panama papers: https://www.theguardian.com/world/2018/mar/14/mossack-fonseca-shut-down-panama-papers https://www.theguardian.com/world/2018/mar/14/mossack-fonsec...
  - datadrivenangel 1y ago
    A PII for children data breach at a Fortune 1000 sized company can easily cost 10s of millions of dollars in employee time to fully resolve.
    - rvz 1y ago
      ...and a massive fine in the millions on top of that if you have customers that are from the EU.
- deleted 1y ago
  [deleted]
- 1y ago
fn-mote 1y ago
The trifecta: > LLM access to untrusted data, the ability to read valuable secrets and the ability to communicate with the outside world The suggestion is to reduce risk by setting boundaries. Seems like security 101.
- danenania 1y ago
  It is, but there's a direct tension here between security and capabilities. It's hard to do useful things with private data without opening up prompt injection holes. And there's a huge demand for this kind of product. Agents also typically work better when you combine all the relevant context as much as possible rather than splitting out and isolating context. See: https://cognition.ai/blog/dont-build-multi-agents https://cognition.ai/blog/dont-build-multi-agents — but this is at odds with isolating agents that read untrusted input.
  - kccqzy 1y ago
    The external communication part of the trifecta is an easy defense. Don't allow external communication. Any external information that's helpful for the AI agent should be available offline, be present in its model (possibly fine tuned).
    - zbentley 1y ago
      Sure, but that is as vacuously true as saying “router keeps getting hacked? Just unplug it from the internet.” Huge numbers of businesses want to use AI in the “hey, watch my inbox and send bills to all the vendors who email me” or “get a count of all the work tickets closed across the company in the last hour and add that to a spreadsheet in sharepoint” variety of automation tasks. Whether those are good ideas or appropriate use-cases for AI is a separate question.
- rvz 1y ago
  It is security 101 as this is just setting basic access controls at the very least. The moment it has access to the internet, the risk is vastly increased. But with a very clever security researcher, it is possible to take over the entire machine with a single prompt injection attack reducing at least one of the requirements.
mellosouls 1y ago
Original @simonw article here: https://simonw.substack.com/p/the-lethal-trifecta-for-ai-agents https://simonw.substack.com/p/the-lethal-trifecta-for-ai-age... https://simonwillison.net/2025/Aug/9/bay-area-ai/ https://simonwillison.net/2025/Aug/9/bay-area-ai/ Discussed: https://news.ycombinator.com/item?id=44846922 https://news.ycombinator.com/item?id=44846922
cobbal 1y ago
Wait, the only way they suggest solving the problem by rate limiting and using a better model? Software engineers figured out these things decades ago. As a field, we already know how to do security. It's just difficult and incompatible with the careless mindset of AI products.
- rvz 1y ago
  > Software engineers figured out these things decades ago. Well this is what happens when a new industry attempts to reinvent poor standards and ignores security best practices just to rush out "AI products" for the sake of it. We have already seen how (flawed) standards like MCPs were hacked immediately from the start and the approaches developers took to "secure" them with somewhat "better prompting" which is just laughable. The worst part of all of this was almost everyone in the AI industry not questioning the security ramifications behind MCP servers having direct access to databases which is a disaster waiting to happen. Just because you can doesn't mean you should and we are seeing how hundreds of AI products are getting breached because of this carelessness in security, even before I mentioned if the product was "vibe coded" or not.
- crazygringo 1y ago
  > As a field, we already know how to do security. Well, AI is part of the field now, so... no, we don't anymore. There's nothing "careless" about AI. The fact that there's no foolproof way to distinguish instruction tokens from data tokens is not careless, it's a fundamental epistemological constraint that human communication suffers from as well. Saying that "software engineers figured out these things decades ago" is deep hubris based on false assumptions.
  - cowboylowrez 1y ago
    >software engineers figured out these things decades ago its true, when engineers fail in this, its called a mistake, and mistakes have consequences unfortunately. If you want to avoid responsibility for mistakes, then llms are the way to go.
  - zbentley 1y ago
simonw 1y ago
This is the second Economist article to mention the lethal trifecta in the past week - the first was https://www.economist.com/science-and-technology/2025/09/22/why-ai-systems-may-never-be-secure-and-what-to-do-about-it https://www.economist.com/science-and-technology/2025/09/22/... - which was the clearest explanations I've seen anywhere in the mainstream media about what prompt injection is and why it's such a nasty threat. (And yeah I got some quotes in it so I may be biased there, but it genuinely is the source I would send executives to in order to understand this.) I like this new one a lot less. It talks about how LLMs are non-deterministic, making them harder to fix security holes in... but then argues that this puts them in the same category as bridges where the solution is to over-engineer them and plan for tolerances and unpredictability. While that's true for the general case of building against LLMs, I don't think it's the right answer for security flaws. If your system only falls victim to 1/100 prompt injection attacks... your system is fundamentally insecure, because an attacker will keep on trying variants of attacks until they find one that works. The way to protect against the lethal trifecta is to cut off one of the legs! If the system doesn't have all three of access to private data, exposure to untrusted instructions and an exfiltration mechanism then the attack doesn't work.
- skrebbel 1y ago
  Must be pretty cool to blog something and post it to a nerd forum like HN and have it picked up by the Economist! Nicely done.
  - simonw 1y ago
    I got to have coffee with their AI/technology editor a few months ago. Having a blog is awesome!
- belter 1y ago
  Love your work. Do you have an opinion on this? "Safeguard your generative AI workloads from prompt injections" - https://aws.amazon.com/blogs/security/safeguard-your-generative-ai-workloads-from-prompt-injections/ https://aws.amazon.com/blogs/security/safeguard-your-generat...
collinmcnulty 1y ago
As a mechanical engineer by background, this article feels weak. Yes it is common to “throw more steel at it” to use a modern version of the sentiment, but that’s still based on knowing in detail the many different ways a structure can fail. The lethal trifecta is a failure mode, you put your “steel” into making sure it doesn’t occur. You would never say “this bridge vibrates violently, how can we make it safe to cross a vibrating bridge”, you’d change the bridge to make it not vibrate out of control.
- switchbak 1y ago
  When a byline starts with "coders need to" I immediately start to tune out. It felt like the analogy was a bit off, and it sounds like that's true to someone with knowledge in the actual domain. "If a company, eager to offer a powerful ai assistant to its employees, gives an LLM access to untrusted data, the ability to read valuable secrets and the ability to communicate with the outside world at the same time" - that's quite the "if", and therein lies the problem. If your company is so enthusiastic to offer functionality that it does so at the cost of security (often knowingly), then you're not taking the situation seriously. And this is a great many companies at present. "Unlike most software, LLMs are probabilistic ... A deterministic approach to safety is thus inadequate" - complete non-sequitur there. Why if a system is non-deterministic is a deterministic approach inadequate? That doesn't even pass the sniff test. That's like saying a virtual machine is inadequate to sandbox a process if the process does non-deterministic things - which is not a sensible argument. As usual, these contrived analogies are taken beyond any reasonable measure and end up making the whole article have very little value. Skipping the analogies and using terminology relevant to the domain would be a good start - but that's probably not as easy to sell to The Economist.
  - semiquaver 1y ago
    > When a byline starts with "coders need to" A byline lists the author of the article. The secondary summary line you’re referring to that appears under the headline is called a “rubric”. https://www.quora.com/Why-does-The-Economist-sometimes-have-separate-headlines-for-the-same-article-on-print-and-digital https://www.quora.com/Why-does-The-Economist-sometimes-have-...
SAI_Peregrinus 1y ago
LLMs don't make a distinction between prompt & data. There's no equivalent to an "NX bit", and AFAIK nobody has figured out how to create such an equivalent. And of course even that wouldn't stop all security issues, just as the NX bit being added to CPUs didn't stop all remote code execution attacks. So the best options we have right now tend to be based around using existing security mechanisms on the LLM agent process. If it runs as a special user then the regular filesystem permissions can restrict its access to various files, and various other mechanisms can be used to restrict access to other resources (outgoing network connections, various hardware, cgroups, etc.). But as long as untrusted data can contain instructions it'll be possible for the LLM output to contain secret data, and if the human using the LLM doesn't notice & copies that output somewhere public the exfiltration step returns.
- boothby 1y ago
  > AFAIK nobody has figured out how to create such an equivalent. I'm curious if anybody has even attempted it; if there's even training data for this. Compartmentalization is a natural aspect of cognition in social creatures. I've even known dogs to not to demonstrate knowledge of a food supply until they think they're not being observed. As a working professional with children, I need to compartmentalize: my social life, sensitive IP knowledge, my kid's private information, knowledge my kid isn't developmentally ready for, my internal thoughts, information I've gained from disreputable sources, and more. Intelligence may be important, but this is wisdom -- something that doesn't seem to be a first-class consideration if dogs and toddlers are in the lead.
techlatest_net 1y ago
[dead]
crazygringo 1y ago
There's an interesting quote from the associated longer article [1]: > In March, researchers at Google proposed a system called CaMeL that uses two separate LLMs to get round some aspects of the lethal trifecta. One has access to untrusted data; the other has access to everything else. The trusted model turns verbal commands from a user into lines of code, with strict limits imposed on them. The untrusted model is restricted to filling in the blanks in the resulting order. This arrangement provides security guarantees, but at the cost of constraining the sorts of tasks the LLMs can perform. This is the first I've heard of it, and seems clever. I'm curious how effective it is. Does it actually provide absolute security guarantees? What sorts of constraints does it have? I'm wondering if this is a real path forward or not. [1] https://www.economist.com/science-and-technology/2025/09/22/why-ai-systems-may-never-be-secure-and-what-to-do-about-it https://www.economist.com/science-and-technology/2025/09/22/...
- simonw 1y ago
  I wrote at length about the CaMeL paper here - I think it's a solid approach but it's also very difficult to implement and greatly restricts what the resulting systems can do: https://simonwillison.net/2025/Apr/11/camel/ https://simonwillison.net/2025/Apr/11/camel/
  - crazygringo 1y ago
    Thank you! That is very helpful. I'm very surprised I haven't come across it on HN before. Seems like CaMeL ought to be a front-page story here... seems like the paper got 16 comments 5 months ago, which isn't much: https://news.ycombinator.com/item?id=43733683 https://news.ycombinator.com/item?id=43733683
jngiam1 1y ago
I have been thinking that the appropriate solution here is to detect when one of the legs is appearing to be a risk and then cutting it off if so. You don’t want to have a blanket policy since that makes it no longer useful, but you want to know when something bad is happening.
deleted 1y ago
[deleted]
neallindsay 1y ago
In-band signaling can never be secure. Doesn't anyone remember the Captain Crunch whistle?
jschveibinz 1y ago
I would think a small subscription fee for a service that automatically monitored (locally) a LLM account (for prompt injections) and provided a filtering pipeline for any inputs to a LLM account would be a software opportunity to investigate.

115 points by 1vuio0pswjnm7 1y ago

"And that means AI engineers need to start thinking like engineers, who build things like bridges and therefore know that shoddy work costs lives." "AI engineers, inculcated in this way of thinking from their schooldays, therefore often act as if problems can be solved just with more training data and more astute system prompts."

dpflan 1y ago
Sounds like suggesting some sort of software engineering board certification plus and ethics certification — the “Von Neumann Oath”? Unethical while still legal software is just extremely lucrative, it seems hard to have this idea take flight.
DaiPlusPlus 1y ago
> can be solved just with more training data Well, y'see - those deaths of innocent people *are* the training data.
roughly 1y ago
> AI engineers need to start thinking like engineers By which they mean actual engineers, not software engineers, who should also probably start thinking like real engineers now that our code’s going into both the bridges and the cars driving over them.
- bubblyworld 1y ago
  What are the kinds of things real engineers do that we could learn from? I hear this a lot ("programmers aren't real engineers") and I'm sympathetic, honestly, but I don't know where to start improving in that regard.
  - skydhash 1y ago
    Act like creating a merge-request to main can expose you to bankruptcy or put you in jail. AKA investigate the impact of a diff to all the failure modes of a software.
  - Mistletoe 1y ago
    What is the factor of safety on your code? https://en.wikipedia.org/wiki/Factor_of_safety https://en.wikipedia.org/wiki/Factor_of_safety
    - bubblyworld 1y ago
      Yeah, I think safety factors and concepts like redundancy have pretty good counterparts in software. Slightly embarrassed to say that I don't know for my current project!
  - roughly 1y ago
    This is off the cuff, but comparing software & software systems to things like buildings, bridges, or real-world infrastructure, there's three broad gaps, I think: 1) We don't have a good sense of the "materials" we're working with - when you're putting up a building, you know the tensile strength of the materials you're working with, how many girders you need to support this much weight/stress, etc. We don't have the same for our systems - every large scale system is effectively designed clean-sheet. We may have prior experience and intuition, but we don't have models, and we can't "prove" our designs ahead of time. 2. Following on the above, we don't have professional standards or certifications. Anyone can call themselves a software engineer, and we don't have a good way of actually testing for competence or knowledge. We don't really do things like apprenticeships or any kind of formalized process of ensuring someone has the set of professional skills required to do something like write the software that's going to be controlling 3 tons of metal moving at 80MPH. 3. We rely too heavily on the ability to patch after the fact - when a bridge or a building requires an update after construction is complete, it's considered a severe fuckup. When a piece of software does, that's normal. By and large, this has historically been fine, because a website going down isn't a huge issue, but when we're talking about things like avionics suites - or even things like Facebook, which is the primary media channel for a large segment of the population - there's real world effects to all the bugs we're fixing in 2.0. Again, by and large most of this has mostly been fine, because the stakes were pretty low, but software's leaked into the real world now, and our "move fast and break things" attitude isn't really compatible with physical objects.
- kevin_thibedeau 1y ago
  Engineering uses repeatable processes to produce expected results. Margin is added to quantifiable elements of a system to reduce the likelihood of failures. You can't add margin on a black box generated by throwing spaghetti at the wall.
  - recursive 1y ago
    You can. We know the properties of materials based on experimentation. In the same way, we can statistically quantify the results that come out of any kind of spaghetti box, based on repeated trials. Just like it's done in many other fields. Science is based on repeated testing of hypotheses. You rarely get black and white answers, just results that suggest things. Like the tensile strength of some particular steel alloy or something.
  - xboxnolifes 1y ago
    Practically everything engineers have to interact with and consider are equivalent to a software black box. Rainfall, winds, tectonic shifts, material properties, etc. Humans don't have the source code to these things. We observe them, we quantify them, notice trends, model the observations, and we apply statistical analysis on them. And it's possible that a real engineer might do all this with an AI model and then determine it's not adequate and choose to not use it.
  - chasd00 1y ago
    > Engineering uses repeatable processes to produce expected results this is the thing with LLMs, the response to a prompt is not guaranteed to be repeatable. Why would you use something like that in an automation where repeatability is required? That's the whole point of automation, repeatability. Would you use a while loop that you can expect to iterate the specified number of times _almost_ every time?
1vuio0pswjnm7 1y ago
In addition to software "engineers", don't forget about software "architects"

throwup238 1y ago

https://archive.ph/8O2aG https://archive.ph/8O2aG

lowbloodsugar 1y ago

Data breaches are hardly lethal. When we’re talking about AI there are plenty of actually lethal failure modes.

HPsquared 1y ago
Depends on the data.
simonw 1y ago
If the breached data is API keys that can be used to rack up charges, it's going to cost you a bunch of money. If it's a crypto wallet then your crypto is irreversibly gone. If the breached data is "material" - i.e. gives someone an advantage in stock market decisions - you're going to get in a lot of trouble with the SEC. If the breached data is PII you're going to get in trouble with all kinds of government agencies. If it's PII for children you're in a world of pain. Update: I found one story about a company going bankrupt after a breach, which is the closest I can get to "lethal": https://www.securityweek.com/amca-files-bankruptcy-following-data-breach/ https://www.securityweek.com/amca-files-bankruptcy-following... Also it turns out Mossack Fonseca shut down after the Panama papers: https://www.theguardian.com/world/2018/mar/14/mossack-fonseca-shut-down-panama-papers https://www.theguardian.com/world/2018/mar/14/mossack-fonsec...
- datadrivenangel 1y ago
  A PII for children data breach at a Fortune 1000 sized company can easily cost 10s of millions of dollars in employee time to fully resolve.
  - rvz 1y ago
    ...and a massive fine in the millions on top of that if you have customers that are from the EU.
deleted 1y ago
[deleted]
1y ago

fn-mote 1y ago

The trifecta: > LLM access to untrusted data, the ability to read valuable secrets and the ability to communicate with the outside world The suggestion is to reduce risk by setting boundaries. Seems like security 101.

danenania 1y ago
It is, but there's a direct tension here between security and capabilities. It's hard to do useful things with private data without opening up prompt injection holes. And there's a huge demand for this kind of product. Agents also typically work better when you combine all the relevant context as much as possible rather than splitting out and isolating context. See: https://cognition.ai/blog/dont-build-multi-agents https://cognition.ai/blog/dont-build-multi-agents — but this is at odds with isolating agents that read untrusted input.
- kccqzy 1y ago
  The external communication part of the trifecta is an easy defense. Don't allow external communication. Any external information that's helpful for the AI agent should be available offline, be present in its model (possibly fine tuned).
  - zbentley 1y ago
    Sure, but that is as vacuously true as saying “router keeps getting hacked? Just unplug it from the internet.” Huge numbers of businesses want to use AI in the “hey, watch my inbox and send bills to all the vendors who email me” or “get a count of all the work tickets closed across the company in the last hour and add that to a spreadsheet in sharepoint” variety of automation tasks. Whether those are good ideas or appropriate use-cases for AI is a separate question.
rvz 1y ago
It is security 101 as this is just setting basic access controls at the very least. The moment it has access to the internet, the risk is vastly increased. But with a very clever security researcher, it is possible to take over the entire machine with a single prompt injection attack reducing at least one of the requirements.

mellosouls 1y ago

Original @simonw article here: https://simonw.substack.com/p/the-lethal-trifecta-for-ai-agents https://simonw.substack.com/p/the-lethal-trifecta-for-ai-age... https://simonwillison.net/2025/Aug/9/bay-area-ai/ https://simonwillison.net/2025/Aug/9/bay-area-ai/ Discussed: https://news.ycombinator.com/item?id=44846922 https://news.ycombinator.com/item?id=44846922

cobbal 1y ago

Wait, the only way they suggest solving the problem by rate limiting and using a better model? Software engineers figured out these things decades ago. As a field, we already know how to do security. It's just difficult and incompatible with the careless mindset of AI products.

rvz 1y ago
> Software engineers figured out these things decades ago. Well this is what happens when a new industry attempts to reinvent poor standards and ignores security best practices just to rush out "AI products" for the sake of it. We have already seen how (flawed) standards like MCPs were hacked immediately from the start and the approaches developers took to "secure" them with somewhat "better prompting" which is just laughable. The worst part of all of this was almost everyone in the AI industry not questioning the security ramifications behind MCP servers having direct access to databases which is a disaster waiting to happen. Just because you can doesn't mean you should and we are seeing how hundreds of AI products are getting breached because of this carelessness in security, even before I mentioned if the product was "vibe coded" or not.
crazygringo 1y ago
> As a field, we already know how to do security. Well, AI is part of the field now, so... no, we don't anymore. There's nothing "careless" about AI. The fact that there's no foolproof way to distinguish instruction tokens from data tokens is not careless, it's a fundamental epistemological constraint that human communication suffers from as well. Saying that "software engineers figured out these things decades ago" is deep hubris based on false assumptions.
- cowboylowrez 1y ago
  >software engineers figured out these things decades ago its true, when engineers fail in this, its called a mistake, and mistakes have consequences unfortunately. If you want to avoid responsibility for mistakes, then llms are the way to go.
- zbentley 1y ago

simonw 1y ago

This is the second Economist article to mention the lethal trifecta in the past week - the first was https://www.economist.com/science-and-technology/2025/09/22/why-ai-systems-may-never-be-secure-and-what-to-do-about-it https://www.economist.com/science-and-technology/2025/09/22/... - which was the clearest explanations I've seen anywhere in the mainstream media about what prompt injection is and why it's such a nasty threat. (And yeah I got some quotes in it so I may be biased there, but it genuinely is the source I would send executives to in order to understand this.) I like this new one a lot less. It talks about how LLMs are non-deterministic, making them harder to fix security holes in... but then argues that this puts them in the same category as bridges where the solution is to over-engineer them and plan for tolerances and unpredictability. While that's true for the general case of building against LLMs, I don't think it's the right answer for security flaws. If your system only falls victim to 1/100 prompt injection attacks... your system is fundamentally insecure, because an attacker will keep on trying variants of attacks until they find one that works. The way to protect against the lethal trifecta is to cut off one of the legs! If the system doesn't have all three of access to private data, exposure to untrusted instructions and an exfiltration mechanism then the attack doesn't work.

skrebbel 1y ago
Must be pretty cool to blog something and post it to a nerd forum like HN and have it picked up by the Economist! Nicely done.
- simonw 1y ago
  I got to have coffee with their AI/technology editor a few months ago. Having a blog is awesome!
belter 1y ago
Love your work. Do you have an opinion on this? "Safeguard your generative AI workloads from prompt injections" - https://aws.amazon.com/blogs/security/safeguard-your-generative-ai-workloads-from-prompt-injections/ https://aws.amazon.com/blogs/security/safeguard-your-generat...

collinmcnulty 1y ago

As a mechanical engineer by background, this article feels weak. Yes it is common to “throw more steel at it” to use a modern version of the sentiment, but that’s still based on knowing in detail the many different ways a structure can fail. The lethal trifecta is a failure mode, you put your “steel” into making sure it doesn’t occur. You would never say “this bridge vibrates violently, how can we make it safe to cross a vibrating bridge”, you’d change the bridge to make it not vibrate out of control.

switchbak 1y ago
When a byline starts with "coders need to" I immediately start to tune out. It felt like the analogy was a bit off, and it sounds like that's true to someone with knowledge in the actual domain. "If a company, eager to offer a powerful ai assistant to its employees, gives an LLM access to untrusted data, the ability to read valuable secrets and the ability to communicate with the outside world at the same time" - that's quite the "if", and therein lies the problem. If your company is so enthusiastic to offer functionality that it does so at the cost of security (often knowingly), then you're not taking the situation seriously. And this is a great many companies at present. "Unlike most software, LLMs are probabilistic ... A deterministic approach to safety is thus inadequate" - complete non-sequitur there. Why if a system is non-deterministic is a deterministic approach inadequate? That doesn't even pass the sniff test. That's like saying a virtual machine is inadequate to sandbox a process if the process does non-deterministic things - which is not a sensible argument. As usual, these contrived analogies are taken beyond any reasonable measure and end up making the whole article have very little value. Skipping the analogies and using terminology relevant to the domain would be a good start - but that's probably not as easy to sell to The Economist.
- semiquaver 1y ago
  > When a byline starts with "coders need to" A byline lists the author of the article. The secondary summary line you’re referring to that appears under the headline is called a “rubric”. https://www.quora.com/Why-does-The-Economist-sometimes-have-separate-headlines-for-the-same-article-on-print-and-digital https://www.quora.com/Why-does-The-Economist-sometimes-have-...

SAI_Peregrinus 1y ago

LLMs don't make a distinction between prompt & data. There's no equivalent to an "NX bit", and AFAIK nobody has figured out how to create such an equivalent. And of course even that wouldn't stop all security issues, just as the NX bit being added to CPUs didn't stop all remote code execution attacks. So the best options we have right now tend to be based around using existing security mechanisms on the LLM agent process. If it runs as a special user then the regular filesystem permissions can restrict its access to various files, and various other mechanisms can be used to restrict access to other resources (outgoing network connections, various hardware, cgroups, etc.). But as long as untrusted data can contain instructions it'll be possible for the LLM output to contain secret data, and if the human using the LLM doesn't notice & copies that output somewhere public the exfiltration step returns.

boothby 1y ago
> AFAIK nobody has figured out how to create such an equivalent. I'm curious if anybody has even attempted it; if there's even training data for this. Compartmentalization is a natural aspect of cognition in social creatures. I've even known dogs to not to demonstrate knowledge of a food supply until they think they're not being observed. As a working professional with children, I need to compartmentalize: my social life, sensitive IP knowledge, my kid's private information, knowledge my kid isn't developmentally ready for, my internal thoughts, information I've gained from disreputable sources, and more. Intelligence may be important, but this is wisdom -- something that doesn't seem to be a first-class consideration if dogs and toddlers are in the lead.

techlatest_net 1y ago

[dead]

crazygringo 1y ago

There's an interesting quote from the associated longer article [1]: > In March, researchers at Google proposed a system called CaMeL that uses two separate LLMs to get round some aspects of the lethal trifecta. One has access to untrusted data; the other has access to everything else. The trusted model turns verbal commands from a user into lines of code, with strict limits imposed on them. The untrusted model is restricted to filling in the blanks in the resulting order. This arrangement provides security guarantees, but at the cost of constraining the sorts of tasks the LLMs can perform. This is the first I've heard of it, and seems clever. I'm curious how effective it is. Does it actually provide absolute security guarantees? What sorts of constraints does it have? I'm wondering if this is a real path forward or not. [1] https://www.economist.com/science-and-technology/2025/09/22/why-ai-systems-may-never-be-secure-and-what-to-do-about-it https://www.economist.com/science-and-technology/2025/09/22/...

simonw 1y ago
I wrote at length about the CaMeL paper here - I think it's a solid approach but it's also very difficult to implement and greatly restricts what the resulting systems can do: https://simonwillison.net/2025/Apr/11/camel/ https://simonwillison.net/2025/Apr/11/camel/
- crazygringo 1y ago
  Thank you! That is very helpful. I'm very surprised I haven't come across it on HN before. Seems like CaMeL ought to be a front-page story here... seems like the paper got 16 comments 5 months ago, which isn't much: https://news.ycombinator.com/item?id=43733683 https://news.ycombinator.com/item?id=43733683

jngiam1 1y ago

I have been thinking that the appropriate solution here is to detect when one of the legs is appearing to be a risk and then cutting it off if so. You don’t want to have a blanket policy since that makes it no longer useful, but you want to know when something bad is happening.

deleted 1y ago

[deleted]

neallindsay 1y ago

In-band signaling can never be secure. Doesn't anyone remember the Captain Crunch whistle?

jschveibinz 1y ago

I would think a small subscription fee for a service that automatically monitored (locally) a LLM account (for prompt injections) and provided a filtering pipeline for any inputs to a LLM account would be a software opportunity to investigate.