7 ms·

Yeah, and as noted over there, this isn't so much an attack. It requires: - you give a system access to your private data - you give an external user access to

by dkdcio 1y ago

Yeah, and as noted over there, this isn't so much an attack. It requires: - you give a system access to your private data - you give an external user access to that system It is hopefully obvious that once you've given an LLM-based system access to some private data and give an external user the ability to input arbitrary text into that system, you've indirectly given the external user access to the private data. This is trivial to solve with standard security best practices.

emidoots 1y ago
Clearly different - but reminds me of the Slack prompt injection vulnerability[0] [0] https://www.theregister.com/2024/08/21/slack_ai_prompt_injection/ https://www.theregister.com/2024/08/21/slack_ai_prompt_injec...
saurik 1y ago
Sure, but like, that's how everyone is using MCP. If your point is that MCP is either fundamentally a bad idea (or was at least fundamentally designed incorrectly) then I agree with you 100%--or if the argument is that a model either isn't smart enough (yet) or aligned enough (maybe ever) to be given access to anything you care about, I also would agree--but, the entire point of this tech is to give models access to private data and then the model is going to, fundamentally to accomplish any goal, see arbitrary text... this is just someone noting "look it isn't even hard to do this" as a reaction to all the people out there (and on here) who want to YOLO this stuff.
- brookst 1y ago
  MCP is a great idea implemented poorly. I shouldn’t have to decide between giving a model access to everything I can access, or nothing. Models should be treated like interns; they are eager and operate in good faith, but they can be fooled, and they can be wrong. MCP says every model is a sysadmin, or at least has the same privileges as the person who hires them. That’s a really bad idea.
  - vel0city 1y ago
    But you don't have to give it everything or nothing. You can just scope the token you give the MCP to the things you want it to access. Even in this instance if they just gave the MCP a token that only had access to this repo (an entirely possible thing to do) it wouldn't have been able to do what it did.
simonw 1y ago
I don't think that's obvious to people at all. I wrote about this one here: https://simonwillison.net/2025/May/26/github-mcp-exploited/ https://simonwillison.net/2025/May/26/github-mcp-exploited/ The key thing people need to understand is what I'm calling the lethal trifecta for prompt injection: access to private data, exposure to malicious instructions and the ability to exfiltrate information. Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. Which means they might be able to abuse its permission to access your private data and have it steal that data on their behalf. "This is trivial to solve with standard security best practices." I don't think that's true. which standard security practices can help here?
- protocolture 1y ago
  Assume that the user has all the privileges of the application (IIRC tricking privileged applications into doing things for you was all the rage in linux privilege escalation attacks back in the day) Apply the principle of least privilege. Either the user doesnt get access to the LLM or the LLM doesnt get access to the tool.
- kiitos 1y ago
  There is no attacker in this situation. In order for the LLM to emit sensitive data publicly, you yourself need to explicitly tell the LLM to evaluate arbitrary third-party input directly, with access to an MCP server you've explicitly defined and configured to have privileged access to your own private information, and then take the output of that response and publish it to a public third-party system without oversight or control. > Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. Whether or not a given tool can be exposed to unverified input from untrusted third-parties is determined by you, not someone else. An attacker can only send you stuff, they can't magically force that stuff to be triggered/processed without your consent.

by dkdcio 1y ago

emidoots 1y ago

Clearly different - but reminds me of the Slack prompt injection vulnerability[0] [0] https://www.theregister.com/2024/08/21/slack_ai_prompt_injection/ https://www.theregister.com/2024/08/21/slack_ai_prompt_injec...

saurik 1y ago

Sure, but like, that's how everyone is using MCP. If your point is that MCP is either fundamentally a bad idea (or was at least fundamentally designed incorrectly) then I agree with you 100%--or if the argument is that a model either isn't smart enough (yet) or aligned enough (maybe ever) to be given access to anything you care about, I also would agree--but, the entire point of this tech is to give models access to private data and then the model is going to, fundamentally to accomplish any goal, see arbitrary text... this is just someone noting "look it isn't even hard to do this" as a reaction to all the people out there (and on here) who want to YOLO this stuff.

brookst 1y ago
MCP is a great idea implemented poorly. I shouldn’t have to decide between giving a model access to everything I can access, or nothing. Models should be treated like interns; they are eager and operate in good faith, but they can be fooled, and they can be wrong. MCP says every model is a sysadmin, or at least has the same privileges as the person who hires them. That’s a really bad idea.
- vel0city 1y ago
  But you don't have to give it everything or nothing. You can just scope the token you give the MCP to the things you want it to access. Even in this instance if they just gave the MCP a token that only had access to this repo (an entirely possible thing to do) it wouldn't have been able to do what it did.

simonw 1y ago

I don't think that's obvious to people at all. I wrote about this one here: https://simonwillison.net/2025/May/26/github-mcp-exploited/ https://simonwillison.net/2025/May/26/github-mcp-exploited/ The key thing people need to understand is what I'm calling the lethal trifecta for prompt injection: access to private data, exposure to malicious instructions and the ability to exfiltrate information. Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. Which means they might be able to abuse its permission to access your private data and have it steal that data on their behalf. "This is trivial to solve with standard security best practices." I don't think that's true. which standard security practices can help here?

protocolture 1y ago
Assume that the user has all the privileges of the application (IIRC tricking privileged applications into doing things for you was all the rage in linux privilege escalation attacks back in the day) Apply the principle of least privilege. Either the user doesnt get access to the LLM or the LLM doesnt get access to the tool.
kiitos 1y ago
There is no attacker in this situation. In order for the LLM to emit sensitive data publicly, you yourself need to explicitly tell the LLM to evaluate arbitrary third-party input directly, with access to an MCP server you've explicitly defined and configured to have privileged access to your own private information, and then take the output of that response and publish it to a public third-party system without oversight or control. > Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. Whether or not a given tool can be exposed to unverified input from untrusted third-parties is determined by you, not someone else. An attacker can only send you stuff, they can't magically force that stuff to be triggered/processed without your consent.