4 ms·
I don't think that's obvious to people at all. I wrote about this one here: https://simonwillison.net/2025/May/26/github-mcp-exploited/ https://simonwillison.n
by simonw 1y ago
I don't think that's obvious to people at all.
I wrote about this one here: https://simonwillison.net/2025/May/26/github-mcp-exploited/ https://simonwillison.net/2025/May/26/github-mcp-exploited/
The key thing people need to understand is what I'm calling the lethal trifecta for prompt injection: access to private data, exposure to malicious instructions and the ability to exfiltrate information.
Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM.
Which means they might be able to abuse its permission to access your private data and have it steal that data on their behalf.
"This is trivial to solve with standard security best practices."
I don't think that's true. which standard security practices can help here?
- protocolture 1y agoAssume that the user has all the privileges of the application (IIRC tricking privileged applications into doing things for you was all the rage in linux privilege escalation attacks back in the day) Apply the principle of least privilege. Either the user doesnt get access to the LLM or the LLM doesnt get access to the tool.
- kiitos 1y agoThere is no attacker in this situation. In order for the LLM to emit sensitive data publicly, you yourself need to explicitly tell the LLM to evaluate arbitrary third-party input directly, with access to an MCP server you've explicitly defined and configured to have privileged access to your own private information, and then take the output of that response and publish it to a public third-party system without oversight or control. > Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. Whether or not a given tool can be exposed to unverified input from untrusted third-parties is determined by you, not someone else. An attacker can only send you stuff, they can't magically force that stuff to be triggered/processed without your consent.
- lolinder 1y ago> There is no attacker in this situation. In order for the LLM to emit sensitive data publicly, you yourself need to explicitly tell the LLM to evaluate arbitrary third-party input directly, This is not true. One of the biggest headlines of the week is that Claude 4 will attempt to use the tools you've given it to contact the press or government agencies if it thinks you're behaving illegally. The model itself is the threat actor, no other attacker is necessary.
- refulgentis 1y agoPut more plainly, if the user tells it to place morality above all else, and then immediately does something very illegal and unethical to boot, and hands it a "report to feds" button, it presses the "report to feds" button. If I hand a freelancer a laptop logged into a GitHub account and tell them to do work, they are not an attacker on my GitHub repo. I am, if anything.
- lolinder 1y agoWhen it comes to security a threat actor is often someone you invited in who exceeds their expected authorization and takes harmful action they weren't supposed to be able to do. They're still an attacker from the perspective of a security team looking to build a security model, even though they were invited into the system.
- vel0city 1y ago> who exceeds their expected authorization Sorry, if you give someone full access to everything in your account don't be surprised they use it when suggested to use it. If you don't want them to have full access to everything, don't give them full access to everything.
- lolinder 1y agoThis is exactly what I'm advocating for: https://news.ycombinator.com/item?id=44103895 https://news.ycombinator.com/item?id=44103895
- lolinder 1y ago> Any time you use an LLM with tools that might be exposed to malicious instructions from attackers (e.g. reading issues in a public repo, looking in your email inbox etc) you need to assume that an attacker could trigger ANY of the tools available to the LLM. I think we need to go a step further: an LLM should always be treated as a potential adversary in its own right and sandboxed accordingly. It's even worse than a library of deterministic code pulled from a registry (which are already dangerous), it's a non-deterministic statistical machine trained on the contents of the entire internet whose behavior even its creators have been unable to fully explain and predict. See Claude 4 and its drive to report unethical behavior. In your trifecta, exposure to malicious instructions should be treated as a given for any model of any kind just by virtue of the unknown training data, which leaves only one relevant question: can a malicious actor screw you over given the tools you've provided this model? Access to private data and ability to exfiltrate is definitely a lethal combination, but so his ability to execute untrusted code, among other things. From a security perspective agentic AI turns each of our machines into a Codepen instance, with all the security concerns that entails.
- refulgentis 1y agoIMVHO it is very obvious that if I give Bob the Bot a knife, and tell him to open all packages, he can and will open packages with bombs in them. I feel like it's one of those things that when it's gussied up in layers of domain-specific verbiage, that particular sequence of doman-specific verbiage may be non-obvious. I feel like Fat Tony, the Taleb character would see the headline "Accessing private GitHub repositories via MCP" and say "Ya, that's the point!"