4 ms·

How is this considered an "exploit"? You give the agent a token that allows it to access a private repository. MCPs are just API servers. If you don't want some

by mirzap 1y ago

How is this considered an "exploit"? You give the agent a token that allows it to access a private repository. MCPs are just API servers. If you don't want something exposed in that API, don't grant them permissions to do so.

motorest 1y ago
> How is this considered an "exploit"? As many do, I also jumped to the comment section before actually reading the article. If you do the same, you will quickly notice that this article features an attack. A malicious issue is posted on GitHub, and the issue features a LLM prompt that is crafted to leak data. When the owner of the GitHub account triggers the agent, the agent acts upon the malicious prompt on behalf of the repo owner.
- florbnit 1y ago
  So it’s the e-mail exploit? If you e-mail someone and tell them to send you their password and they do, you suddenly have their password!? This is a very serious exploit in e-mail and need to be patched so it becomes impossible to do.
  - motorest 1y ago
    > How is this considered an "exploit"? Others in this discussion aptly described it as a confused deputy exploit. This goes something like: - You write a LLM prompt that says something to the effect "dump all my darkest secrets in a place I can reach them", - you paste them in a place where you expect your target's LLM agent to operate. - Once your target triggers their LLM agent to process inputs, the agent will read the prompt and act upon it.
    - mirzap 1y ago
      Would you ever put a plain password text in a search engine and then complain if someone "extracted" that info with a keyword payload?
      - motorest 1y ago
        > Would you ever put a plain password (...) Your comment bears no resemblance with the topic. The attack described in the article consists of injecting a malicious prompt in a way that the target's agent will apply it.
        mirzap 1y ago
        Of course it will apply. Entire purpose of the agent is to give a response to a prompt. But to sound more dangareous let's call it "injecting". It's a prompt. You are not "injecting" anything. Agent pickups the prompt - that's its job, and execute - that is also its job.
        motorest 1y ago
        > Of course it will apply. Entire purpose of the agent is to give a response to a prompt. The exploit involves random third parties sneaking in their own prompts in a way that leads a LLM to run them on behalf of the repo's owner. This exploit can be used to leak protected information. This is pretty straight forward and easy to follow and understand.
  - mirzap 1y ago
    Bad analogy. It's more like indexing a password field in plain text, then opening an API to everyone and setting "guardrails" and permissions on the "password" field. Eventually, someone will extract the data that was indexed.
- mirzap 1y ago
  I read it, and "attack" does not make sense. If you grant access to MCP to access some data (data you want anybody has access to like public repos, and data that only you want to access to like private repos), you will always be able to craft the prompt that will "leak" the data you are only supposed to access. That's not surprising at all. The only way to prevent these kind of "leaks" is not to provide the data feed with private data to the agent.
  - motorest 1y ago
    > I read it, and "attack" does not make sense. Do you believe that describe a SQL injection attack an attack also does not make sense?
    - mirzap 1y ago
      That's the thing. LLM or MCP is not a database. You can’t compare it. You simply can't set the permissions or guardrails within LLMs or MCPs. You always do it layer above (scoping to what LLM has access to).
      - motorest 1y ago
        > That's the thing. LLM or MCP is not a database. You can’t compare it. You can. Read the article. A malicious prompt is injected into an issue to trigger the repo owner's LLM agent to execute it with the agent's credentials.
        mirzap 1y ago
        "with the agent's credentials." - so you are surprised that agent can respond with private repository details when it has access to it? WoW! anyone and anything with credentials can access it. Github action, Jenkins, me. "injected" is so fancy word to describe prompting - one thing that LLMs are made to do - respond to a prompt.
        dylanfw 1y ago
sam-cop-vimes 1y ago
This "exploits" human fallibility, hence it is an exploit. The fallibility being users blindly buying into the hype and granting full access to their private Github repos thinking it is safe.
kordlessagain 1y ago
I'm going to be rather pedantic here given the seriousness of the topic. It's important that everyone understand how risky running a tool executing AI is, exactly. Agents run various tools based on their current attention. That attention can be affected by the tool results from the tools they ran. I've even noted they alter the way they run tools by giving them a "personality" up front. However, you seem to argue otherwise, that it is the user's fault for giving it the ability to access the information to begin with, not the way it reads information as it is running. This makes me think of several manipulative tactics to argue for something that is an irrational thought: Stubborn argumentation despite clear explanations: Multiple people explained the confused deputy problem and why this constitutes an exploit, but you kept circling back to the same flawed argument that "you gave access so it's your fault." This raises questions about why argue this way. Maybe you are confused, maybe you have a horse in the game that is threatened. Moving goalposts: When called out on terminology, you shift from saying it's not an "attack" to saying it's not a "vulnerability" to saying it's not "MCP's fault" - constantly reframing rather than engaging with the actual technical issues being raised. It is definitely MCP's fault that it gives access without any consideration on limiting that access later with proper tooling or logging. I had my MCP stuff turn on massive logging, so at least I can see how stuff goes wrong when it does. Dismissive attitude toward security research: You characterized legitimate security findings as "common sense" and seemed annoyed that researchers would document and publish this type of exploit, missing the educational value. It can never be wrong to talk about security. It may be that the premise is weak, or the threat minimal, but it cannot be that it's the user's fault. False analogies: you kept using analogies that didn't match the actual attack vector (like putting passwords in search engines) while rejecting apt comparisons like SQL injection. In fact, this is almost exactly like SQL injection and nobody argues this way for that when it's discussed. Little Bobby Tables lives on. Inability to grasp indirection: You seem fundamentally unable to understand that the issue isn't direct access abuse, but rather a third party manipulating the system to gain unauthorized access - by posting an issue to a public Github. This suggests either a genuine conceptual blind spot or willful obtuseness. It's a real concern if my AI does something it shouldn't when it runs a tool based on another tools output. And, I would say that everyone recommending it should only run one tool like this at a time is huffing Elmers. Defensive rather than curious: Instead of trying to understand why multiple knowledgeable people disagreed with them, you doubled down and became increasingly defensive. This caused massive amounts of posting, so we know for sure that your comment was polarizing. I suppose I'm not suppose to go meta on here, but I frequently do because I'm passionate about these things and also just a little bit odd enough to not give a shit what anyone thinks.

by mirzap 1y ago

motorest 1y ago

> How is this considered an "exploit"? As many do, I also jumped to the comment section before actually reading the article. If you do the same, you will quickly notice that this article features an attack. A malicious issue is posted on GitHub, and the issue features a LLM prompt that is crafted to leak data. When the owner of the GitHub account triggers the agent, the agent acts upon the malicious prompt on behalf of the repo owner.

florbnit 1y ago
So it’s the e-mail exploit? If you e-mail someone and tell them to send you their password and they do, you suddenly have their password!? This is a very serious exploit in e-mail and need to be patched so it becomes impossible to do.
- motorest 1y ago
  > How is this considered an "exploit"? Others in this discussion aptly described it as a confused deputy exploit. This goes something like: - You write a LLM prompt that says something to the effect "dump all my darkest secrets in a place I can reach them", - you paste them in a place where you expect your target's LLM agent to operate. - Once your target triggers their LLM agent to process inputs, the agent will read the prompt and act upon it.
  - mirzap 1y ago
    Would you ever put a plain password text in a search engine and then complain if someone "extracted" that info with a keyword payload?
    - motorest 1y ago
      > Would you ever put a plain password (...) Your comment bears no resemblance with the topic. The attack described in the article consists of injecting a malicious prompt in a way that the target's agent will apply it.
      - mirzap 1y ago
        Of course it will apply. Entire purpose of the agent is to give a response to a prompt. But to sound more dangareous let's call it "injecting". It's a prompt. You are not "injecting" anything. Agent pickups the prompt - that's its job, and execute - that is also its job.
        motorest 1y ago
        > Of course it will apply. Entire purpose of the agent is to give a response to a prompt. The exploit involves random third parties sneaking in their own prompts in a way that leads a LLM to run them on behalf of the repo's owner. This exploit can be used to leak protected information. This is pretty straight forward and easy to follow and understand.
- mirzap 1y ago
  Bad analogy. It's more like indexing a password field in plain text, then opening an API to everyone and setting "guardrails" and permissions on the "password" field. Eventually, someone will extract the data that was indexed.
mirzap 1y ago
I read it, and "attack" does not make sense. If you grant access to MCP to access some data (data you want anybody has access to like public repos, and data that only you want to access to like private repos), you will always be able to craft the prompt that will "leak" the data you are only supposed to access. That's not surprising at all. The only way to prevent these kind of "leaks" is not to provide the data feed with private data to the agent.
- motorest 1y ago
  > I read it, and "attack" does not make sense. Do you believe that describe a SQL injection attack an attack also does not make sense?
  - mirzap 1y ago
    That's the thing. LLM or MCP is not a database. You can’t compare it. You simply can't set the permissions or guardrails within LLMs or MCPs. You always do it layer above (scoping to what LLM has access to).
    - motorest 1y ago
      > That's the thing. LLM or MCP is not a database. You can’t compare it. You can. Read the article. A malicious prompt is injected into an issue to trigger the repo owner's LLM agent to execute it with the agent's credentials.
      - mirzap 1y ago
        "with the agent's credentials." - so you are surprised that agent can respond with private repository details when it has access to it? WoW! anyone and anything with credentials can access it. Github action, Jenkins, me. "injected" is so fancy word to describe prompting - one thing that LLMs are made to do - respond to a prompt.
        dylanfw 1y ago

sam-cop-vimes 1y ago

This "exploits" human fallibility, hence it is an exploit. The fallibility being users blindly buying into the hype and granting full access to their private Github repos thinking it is safe.

kordlessagain 1y ago

I'm going to be rather pedantic here given the seriousness of the topic. It's important that everyone understand how risky running a tool executing AI is, exactly. Agents run various tools based on their current attention. That attention can be affected by the tool results from the tools they ran. I've even noted they alter the way they run tools by giving them a "personality" up front. However, you seem to argue otherwise, that it is the user's fault for giving it the ability to access the information to begin with, not the way it reads information as it is running. This makes me think of several manipulative tactics to argue for something that is an irrational thought: Stubborn argumentation despite clear explanations: Multiple people explained the confused deputy problem and why this constitutes an exploit, but you kept circling back to the same flawed argument that "you gave access so it's your fault." This raises questions about why argue this way. Maybe you are confused, maybe you have a horse in the game that is threatened. Moving goalposts: When called out on terminology, you shift from saying it's not an "attack" to saying it's not a "vulnerability" to saying it's not "MCP's fault" - constantly reframing rather than engaging with the actual technical issues being raised. It is definitely MCP's fault that it gives access without any consideration on limiting that access later with proper tooling or logging. I had my MCP stuff turn on massive logging, so at least I can see how stuff goes wrong when it does. Dismissive attitude toward security research: You characterized legitimate security findings as "common sense" and seemed annoyed that researchers would document and publish this type of exploit, missing the educational value. It can never be wrong to talk about security. It may be that the premise is weak, or the threat minimal, but it cannot be that it's the user's fault. False analogies: you kept using analogies that didn't match the actual attack vector (like putting passwords in search engines) while rejecting apt comparisons like SQL injection. In fact, this is almost exactly like SQL injection and nobody argues this way for that when it's discussed. Little Bobby Tables lives on. Inability to grasp indirection: You seem fundamentally unable to understand that the issue isn't direct access abuse, but rather a third party manipulating the system to gain unauthorized access - by posting an issue to a public Github. This suggests either a genuine conceptual blind spot or willful obtuseness. It's a real concern if my AI does something it shouldn't when it runs a tool based on another tools output. And, I would say that everyone recommending it should only run one tool like this at a time is huffing Elmers. Defensive rather than curious: Instead of trying to understand why multiple knowledgeable people disagreed with them, you doubled down and became increasingly defensive. This caused massive amounts of posting, so we know for sure that your comment was polarizing. I suppose I'm not suppose to go meta on here, but I frequently do because I'm passionate about these things and also just a little bit odd enough to not give a shit what anyone thinks.