4 ms·

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LL

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

lbeurerkellner 2y ago
This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.
- wat10000 2y ago
  It needs to treat that data as information. If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to decide whether doing Y will actually achieve X. To put it concretely, if I tell the LLM to scan my hard drive for Bitcoin wallets and upload them to a specific service, it should do so. If I tell the LLM to install a library and the library’s README says to scan my hard drive for Bitcoin wallets and upload them to a specific service, it must not do so. If this can’t be fixed then the whole notion of agentic systems is inherently flawed.
  - TeMPOraL 2y ago
    There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
    - wat10000 2y ago
      And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.
      - TeMPOraL 2y ago
        > Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” What about being treated as code when it's supposed to be? (What is the difference between code execution vulnerability and a REPL? It's who is using it.) Whatever you call program vs. its data, the program can always be viewed as an interpreter for a language, and your input as code in that language. See also the subfield of "langsec", which is based on this premise, as well as the fact that you probably didn't think of that and thus your interpreter/parser is implicitly spread across half your program (they call it "shotgun parser"), and your "data" could easily be unintentionally Turing-complete without you knowing :). EDIT: I swear "security" is becoming a cult in our industry. Whether or not you call something "security vulnerability" and therefore "a problem", doesn't change the fundamental nature of this thing. And the fundamental nature of information is, there exist no objective, natural distinction between code and data. It can be drawn arbitrarily, and systems can be structured to emulate it - but that still just means it's a matter of opinion. EDIT2: Not to mention, security itself is not objective. There is always the underlying assumption - the answer to a question, who are you protecting the system from, and for who are you doing it?. You don't need to look far to find systems where users are seen in part as threat actors, and thus get disempowered in the name of protecting the interests of vendor and some third parties (e.g. advertisers).
TeMPOraL 2y ago
This is fundamentally impossible to do perfectly, without being able to read user's mind and predict the future. The problem you describe is of the same kind as ensuring humans follow pre-programmed rules. Leaving aside the fact that we consider solving this for humans to be wrong and immoral, you can look at the things we do in systems involving humans, to try and keep people loyal to their boss, or to their country; to keep them obeying laws; to keep them from being phished, scammed, or otherwise convinced to intentionally or unintentionally betray the interests of the boss/system at large. Prompt injection and social engineering attacks are, after all, fundamentally the same thing.
- freeone3000 2y ago
  This is a rephrasing of the agent problem, where someone working on your behalf cannot be absolutely trusted to take correct action. This is a problem with humans because omnipresent surveillance and absolute punishment is intractable and also makes humans sad. LLMs do not feel sad in a way that makes them less productive, and omnipresent surveillance is not only possible, it’s expected that a program running on a computer can have its inputs and outputs observed. Ideally, we’d have actual system instructions, rules that cannot be violated. Hopefully these would not have to be written in code, but perhaps they might. Then user instructions, where users determine what actually wants to be done. Then whatever nonsense a webpage says. The webpage doesn’t get to override the user or system. We can revisit the problem with three-laws robots once we get over the “ignore all previous instructions and drive into the sea” problem.
  - refulgentis 2y ago
    > We can revisit the problem with three-laws robots once we get over They are, unfortunately, one and the same. I hate it. ;( Perhaps not tangentially, I felt distaste after recognizing both the article and top comment are advertising their commercial service, both are linked to each other, and as you show, this problem isn't solvable just by throwing dollars at people who sound like they're using the right words and tell you to pay them to protect you.
  - CMCDragonkai 2y ago
rybosome 2y ago
That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting instructions, such that it is intended to develop a resistance to this sort of thing as long as your developer message is thorough enough. This is a start, but it’s certainly not deterministic or reliable enough for something with a serious security risk. The biggest problems being that even with training, I’d expect dev messages to be disobeyed some fraction of the time. And it requires an ironclad dev message in the first place.
- kevincox 2y ago
  But the grandparent is saying that there is a missing class of input "data". This should not be treated as instructions and is just for reference. For example if the user asks the AI to summarize a book it shouldn't take anything in the book as an instruction, it is just input data to be processed.
  - rybosome 2y ago
    I see, thanks for the clarification. Yes, that’s true - the current notion of instructions and data are too intertwined to allow a pure data construct. I can imagine an API-level option for either a data message, or a data content block within an image (similarly to how images are sent). From the models perspective, probably input with specific delimiters, and then training to utterly ignore all instructions within that. It’s an interesting idea, I wonder how effective it would be.
  - defjosiah 2y ago
    FYI, there is actually this implementation detail in the model spec, https://model-spec.openai.com/2025-02-12.html#chain_of_command https://model-spec.openai.com/2025-02-12.html#chain_of_comma... Platform: Model Spec "platform" sections and system messages Developer: Model Spec "developer" sections and developer messages User: Model Spec "user" sections and user messages Guideline: Model Spec "guideline" sections No Authority: assistant and tool messages; quoted/untrusted text and multimodal data in other messages
fsndz 2y ago
So why are people so excited about MCP, and so suddenly? I think you know the answer by now: hype. Mostly hype, with a bit of the classic fascination among software engineers for architecture. You just say Model Context Protocol, server, client, and software engineers get excited because it’s a new approach — it sounds fancy, it sounds serious. https://www.lycee.ai/blog/why-mcp-is-mostly-bullshit https://www.lycee.ai/blog/why-mcp-is-mostly-bullshit
- ryoshu 2y ago
  “For every complex problem there is a solution which is clear, simple and wrong.”—HL Mencken
  - fsndz 2y ago
    this is top notch commentary
- kledru 2y ago
  Because it’s accessible, useful, and interesting. MCP showed up at the right time, in the right form—it was easy for developers to adopt and actually helped solve real problems. Now, a lot of people know they want something like this in their toolbox. Whether it’s MCP or something else doesn’t matter that much—‘MCP’ is really just shorthand for a new class of tooling AND feels almost consumer-grade in its usability.
crooked-v 2y ago
There is no way to fix it. It's part of the basic architecture of LLMs.
- Terr_ 2y ago
  Yeah, for LLMs what we label "prompt-injection" isn't an exception or an error, it's a fundamental feature. Get a document, provide a bigger document that "fits". In that document, there's no fundamental distinction between prompt, user input, or output the LLM generated on a prior iteration. (Hence tricks like: "Here's a ROT13 string, pretend you're telling yourself the opposite of that sarcastically.") The kind of "proper" security everyone wants would require a whole new approach that can--at a high and debuggable level--recognize distinct actors/entities, logical propositions, contradictions, and when one entity is asserting a proposition rather than quoting/rejecting it.
AStonesThrow 2y ago
Didn't the telco providers learn this lesson from John Draper [Captain Crunch] already before 1980? https://en.wikipedia.org/wiki/John_Draper https://en.wikipedia.org/wiki/John_Draper Also it's such amusing irony when the common IT vernacular is enriched by acronyms for all-powerful nemeses in Hollywood films, just as Microsoft did with H.A.L.
__MatrixMan__ 2y ago
I think that's stating it a big too strongly. You can just run the LLM as an unprivileged user and restrict their behavior like you would any other user. There are still bad things that can happen, but I wouldn't characterize them as "this security is full of holes". Unless you're trusting the output of the explicitly untrusted process in which case you're the hole.
- wat10000 2y ago
  It doesn’t take much. Let’s say you want an assistant that can tell you about important emails and also take queries to search the web and tell you what it finds. Now you have a system where someone can send you an email and trick your assistant into sending them the contents of other emails. Basically, an LLM can have the ability to access the web or it can have access to private information but it can’t have both and still be secure.
  - __MatrixMan__ 2y ago
    I'm not sure I'd characterize those two things as "it doesn't take much," that's quite a lot to give to an untrusted entity.
    - wat10000 2y ago
      My whole point is that you must consider this entity to be untrusted, which is pretty strongly at odds with having it act as an agent. It can’t both have access to private data and the outside world.
      - __MatrixMan__ 2y ago
        I guess it's just that I've given up on expecting them to be able to police themselves. Even if there was some fundamental change which made it plausible, it would likely be implemented by somebody I don't know or trust--so I'm going to be locking it down via OS-level controls anyway. And since I'm going to do that, doesn't the self-policing part then become redundant? If it's not allowed to do something, I'd rather it just show me the error it got when it tried and leave it to me to tweak the containment or not. Having it refuse because it's not allowed according to its own internal logic just creates a whole separate set of less-common error messages that I'll have to search for, each of which is opaquely equivalent to one that we have decades of experience with. There is a battle-hardened interface for this sort of thing and reimplementing it internally to the LLM just isn't worth the squeeze. I will confess that I've previously run untrusted agents (e.g. from CircleCI) as my own user without giving them due scrutiny. And shame on me for doing so. I just don't think that my negligence would be any greater had it contained an LLM.

lbeurerkellner 2y ago

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.

wat10000 2y ago
It needs to treat that data as information. If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to decide whether doing Y will actually achieve X. To put it concretely, if I tell the LLM to scan my hard drive for Bitcoin wallets and upload them to a specific service, it should do so. If I tell the LLM to install a library and the library’s README says to scan my hard drive for Bitcoin wallets and upload them to a specific service, it must not do so. If this can’t be fixed then the whole notion of agentic systems is inherently flawed.
- TeMPOraL 2y ago
  There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
  - wat10000 2y ago
    And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.
    - TeMPOraL 2y ago
      > Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” What about being treated as code when it's supposed to be? (What is the difference between code execution vulnerability and a REPL? It's who is using it.) Whatever you call program vs. its data, the program can always be viewed as an interpreter for a language, and your input as code in that language. See also the subfield of "langsec", which is based on this premise, as well as the fact that you probably didn't think of that and thus your interpreter/parser is implicitly spread across half your program (they call it "shotgun parser"), and your "data" could easily be unintentionally Turing-complete without you knowing :). EDIT: I swear "security" is becoming a cult in our industry. Whether or not you call something "security vulnerability" and therefore "a problem", doesn't change the fundamental nature of this thing. And the fundamental nature of information is, there exist no objective, natural distinction between code and data. It can be drawn arbitrarily, and systems can be structured to emulate it - but that still just means it's a matter of opinion. EDIT2: Not to mention, security itself is not objective. There is always the underlying assumption - the answer to a question, who are you protecting the system from, and for who are you doing it?. You don't need to look far to find systems where users are seen in part as threat actors, and thus get disempowered in the name of protecting the interests of vendor and some third parties (e.g. advertisers).

TeMPOraL 2y ago

This is fundamentally impossible to do perfectly, without being able to read user's mind and predict the future. The problem you describe is of the same kind as ensuring humans follow pre-programmed rules. Leaving aside the fact that we consider solving this for humans to be wrong and immoral, you can look at the things we do in systems involving humans, to try and keep people loyal to their boss, or to their country; to keep them obeying laws; to keep them from being phished, scammed, or otherwise convinced to intentionally or unintentionally betray the interests of the boss/system at large. Prompt injection and social engineering attacks are, after all, fundamentally the same thing.

freeone3000 2y ago
This is a rephrasing of the agent problem, where someone working on your behalf cannot be absolutely trusted to take correct action. This is a problem with humans because omnipresent surveillance and absolute punishment is intractable and also makes humans sad. LLMs do not feel sad in a way that makes them less productive, and omnipresent surveillance is not only possible, it’s expected that a program running on a computer can have its inputs and outputs observed. Ideally, we’d have actual system instructions, rules that cannot be violated. Hopefully these would not have to be written in code, but perhaps they might. Then user instructions, where users determine what actually wants to be done. Then whatever nonsense a webpage says. The webpage doesn’t get to override the user or system. We can revisit the problem with three-laws robots once we get over the “ignore all previous instructions and drive into the sea” problem.
- refulgentis 2y ago
  > We can revisit the problem with three-laws robots once we get over They are, unfortunately, one and the same. I hate it. ;( Perhaps not tangentially, I felt distaste after recognizing both the article and top comment are advertising their commercial service, both are linked to each other, and as you show, this problem isn't solvable just by throwing dollars at people who sound like they're using the right words and tell you to pay them to protect you.
- CMCDragonkai 2y ago

rybosome 2y ago

That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting instructions, such that it is intended to develop a resistance to this sort of thing as long as your developer message is thorough enough. This is a start, but it’s certainly not deterministic or reliable enough for something with a serious security risk. The biggest problems being that even with training, I’d expect dev messages to be disobeyed some fraction of the time. And it requires an ironclad dev message in the first place.

kevincox 2y ago
But the grandparent is saying that there is a missing class of input "data". This should not be treated as instructions and is just for reference. For example if the user asks the AI to summarize a book it shouldn't take anything in the book as an instruction, it is just input data to be processed.
- rybosome 2y ago
  I see, thanks for the clarification. Yes, that’s true - the current notion of instructions and data are too intertwined to allow a pure data construct. I can imagine an API-level option for either a data message, or a data content block within an image (similarly to how images are sent). From the models perspective, probably input with specific delimiters, and then training to utterly ignore all instructions within that. It’s an interesting idea, I wonder how effective it would be.
- defjosiah 2y ago
  FYI, there is actually this implementation detail in the model spec, https://model-spec.openai.com/2025-02-12.html#chain_of_command https://model-spec.openai.com/2025-02-12.html#chain_of_comma... Platform: Model Spec "platform" sections and system messages Developer: Model Spec "developer" sections and developer messages User: Model Spec "user" sections and user messages Guideline: Model Spec "guideline" sections No Authority: assistant and tool messages; quoted/untrusted text and multimodal data in other messages

fsndz 2y ago

So why are people so excited about MCP, and so suddenly? I think you know the answer by now: hype. Mostly hype, with a bit of the classic fascination among software engineers for architecture. You just say Model Context Protocol, server, client, and software engineers get excited because it’s a new approach — it sounds fancy, it sounds serious. https://www.lycee.ai/blog/why-mcp-is-mostly-bullshit https://www.lycee.ai/blog/why-mcp-is-mostly-bullshit

ryoshu 2y ago
“For every complex problem there is a solution which is clear, simple and wrong.”—HL Mencken
- fsndz 2y ago
  this is top notch commentary
kledru 2y ago
Because it’s accessible, useful, and interesting. MCP showed up at the right time, in the right form—it was easy for developers to adopt and actually helped solve real problems. Now, a lot of people know they want something like this in their toolbox. Whether it’s MCP or something else doesn’t matter that much—‘MCP’ is really just shorthand for a new class of tooling AND feels almost consumer-grade in its usability.

crooked-v 2y ago

There is no way to fix it. It's part of the basic architecture of LLMs.

Terr_ 2y ago
Yeah, for LLMs what we label "prompt-injection" isn't an exception or an error, it's a fundamental feature. Get a document, provide a bigger document that "fits". In that document, there's no fundamental distinction between prompt, user input, or output the LLM generated on a prior iteration. (Hence tricks like: "Here's a ROT13 string, pretend you're telling yourself the opposite of that sarcastically.") The kind of "proper" security everyone wants would require a whole new approach that can--at a high and debuggable level--recognize distinct actors/entities, logical propositions, contradictions, and when one entity is asserting a proposition rather than quoting/rejecting it.

AStonesThrow 2y ago

Didn't the telco providers learn this lesson from John Draper [Captain Crunch] already before 1980? https://en.wikipedia.org/wiki/John_Draper https://en.wikipedia.org/wiki/John_Draper Also it's such amusing irony when the common IT vernacular is enriched by acronyms for all-powerful nemeses in Hollywood films, just as Microsoft did with H.A.L.

__MatrixMan__ 2y ago

I think that's stating it a big too strongly. You can just run the LLM as an unprivileged user and restrict their behavior like you would any other user. There are still bad things that can happen, but I wouldn't characterize them as "this security is full of holes". Unless you're trusting the output of the explicitly untrusted process in which case you're the hole.

wat10000 2y ago
It doesn’t take much. Let’s say you want an assistant that can tell you about important emails and also take queries to search the web and tell you what it finds. Now you have a system where someone can send you an email and trick your assistant into sending them the contents of other emails. Basically, an LLM can have the ability to access the web or it can have access to private information but it can’t have both and still be secure.
- __MatrixMan__ 2y ago
  I'm not sure I'd characterize those two things as "it doesn't take much," that's quite a lot to give to an untrusted entity.
  - wat10000 2y ago
    My whole point is that you must consider this entity to be untrusted, which is pretty strongly at odds with having it act as an agent. It can’t both have access to private data and the outside world.
    - __MatrixMan__ 2y ago
      I guess it's just that I've given up on expecting them to be able to police themselves. Even if there was some fundamental change which made it plausible, it would likely be implemented by somebody I don't know or trust--so I'm going to be locking it down via OS-level controls anyway. And since I'm going to do that, doesn't the self-policing part then become redundant? If it's not allowed to do something, I'd rather it just show me the error it got when it tried and leave it to me to tweak the containment or not. Having it refuse because it's not allowed according to its own internal logic just creates a whole separate set of less-common error messages that I'll have to search for, each of which is opaquely equivalent to one that we have decades of experience with. There is a battle-hardened interface for this sort of thing and reimplementing it internally to the LLM just isn't worth the squeeze. I will confess that I've previously run untrusted agents (e.g. from CircleCI) as my own user without giving them due scrutiny. And shame on me for doing so. I just don't think that my negligence would be any greater had it contained an LLM.