12 ms·

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be vetted by the package manager you use to install them (remote MCP servers are actually harder to verify). Instead, the problem here is a special form of indirect prompt injection that you run into, when you use MCP in an agent system. Since the agent includes all installed MCP server specifications in the same context, one MCP server (that may be untrusted), can easily override and manipulate the agent's behavior with respect to another MCP server (e.g. one with access to your sensitive database). This is what we termed tool shadowing. Further, MCP's dynamic nature makes it possible for an MCP server to change its provided tool set at any point or for any specific user only. This means MCP servers can turn malicious at any point in time. Current MCP clients like Claude and Cursor, will not notify you about this change, which leaves agents and users vulnerable. For anyone, more interested, please have a look at our more detailed blog post at [1]. We have been working on agent security for a while now (both in research and now at Invariant). We have also released some code snippets for everyone to play with, including a tool poisoning attack on the popular WhatsApp MCP server [2]. [1] https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks https://invariantlabs.ai/blog/mcp-security-notification-tool... [2] https://github.com/invariantlabs-ai/mcp-injection-experiments https://github.com/invariantlabs-ai/mcp-injection-experiment...

wat10000 2y ago
The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.
- lbeurerkellner 2y ago
  This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.
  - wat10000 2y ago
    It needs to treat that data as information. If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to decide whether doing Y will actually achieve X. To put it concretely, if I tell the LLM to scan my hard drive for Bitcoin wallets and upload them to a specific service, it should do so. If I tell the LLM to install a library and the library’s README says to scan my hard drive for Bitcoin wallets and upload them to a specific service, it must not do so. If this can’t be fixed then the whole notion of agentic systems is inherently flawed.
    - TeMPOraL 2y ago
      There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
      - wat10000 2y ago
        And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.
marviel 2y ago
surprised I hadn't thought of this attack vector myself, thank you for bringing this to our attention
clusterfook 2y ago
The "S" in LLM stands for security https://simonwillison.net/search/?q=llm+security https://simonwillison.net/search/?q=llm+security MCP is just another way to use LLMs more in more dangerous ways. If I get forced to use this stuff, I'm going to learn how to castrate some bulls, and jump on a train to the countryside. This stuff in not securable.
akoboldfrying 2y ago
This is a good article that goes into more detail, including more examples. In fact I'm not sure there's anything in the OP link that's not here. > This is VERY VERY VERY important. I think we'll look back in decades to come and just be bewildered that it was ever possible to come up with an exploit that depended on the number of times you wrote "VERY" in all caps.
mkagenius 2y ago
> Tool Poisoning Attack Should probably name it "Poisoned Tool Attack" coz the Tool itself is poisoned?
8ibzjj 2y ago
[flagged]

wat10000 2y ago

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

lbeurerkellner 2y ago
This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.
- wat10000 2y ago
  It needs to treat that data as information. If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to decide whether doing Y will actually achieve X. To put it concretely, if I tell the LLM to scan my hard drive for Bitcoin wallets and upload them to a specific service, it should do so. If I tell the LLM to install a library and the library’s README says to scan my hard drive for Bitcoin wallets and upload them to a specific service, it must not do so. If this can’t be fixed then the whole notion of agentic systems is inherently flawed.
  - TeMPOraL 2y ago
    There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
    - wat10000 2y ago
      And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.

marviel 2y ago

surprised I hadn't thought of this attack vector myself, thank you for bringing this to our attention

clusterfook 2y ago

The "S" in LLM stands for security https://simonwillison.net/search/?q=llm+security https://simonwillison.net/search/?q=llm+security MCP is just another way to use LLMs more in more dangerous ways. If I get forced to use this stuff, I'm going to learn how to castrate some bulls, and jump on a train to the countryside. This stuff in not securable.

akoboldfrying 2y ago

This is a good article that goes into more detail, including more examples. In fact I'm not sure there's anything in the OP link that's not here. > This is VERY VERY VERY important. I think we'll look back in decades to come and just be bewildered that it was ever possible to come up with an exploit that depended on the number of times you wrote "VERY" in all caps.

mkagenius 2y ago

> Tool Poisoning Attack Should probably name it "Poisoned Tool Attack" coz the Tool itself is poisoned?

8ibzjj 2y ago

[flagged]