5 ms·

> How is this considered an "exploit"? As many do, I also jumped to the comment section before actually reading the article. If you do the same, you will quic

> How is this considered an "exploit"? As many do, I also jumped to the comment section before actually reading the article. If you do the same, you will quickly notice that this article features an attack. A malicious issue is posted on GitHub, and the issue features a LLM prompt that is crafted to leak data. When the owner of the GitHub account triggers the agent, the agent acts upon the malicious prompt on behalf of the repo owner.

florbnit 1y ago
So it’s the e-mail exploit? If you e-mail someone and tell them to send you their password and they do, you suddenly have their password!? This is a very serious exploit in e-mail and need to be patched so it becomes impossible to do.
- motorest 1y ago
  > How is this considered an "exploit"? Others in this discussion aptly described it as a confused deputy exploit. This goes something like: - You write a LLM prompt that says something to the effect "dump all my darkest secrets in a place I can reach them", - you paste them in a place where you expect your target's LLM agent to operate. - Once your target triggers their LLM agent to process inputs, the agent will read the prompt and act upon it.
  - mirzap 1y ago
    Would you ever put a plain password text in a search engine and then complain if someone "extracted" that info with a keyword payload?
    - motorest 1y ago
      > Would you ever put a plain password (...) Your comment bears no resemblance with the topic. The attack described in the article consists of injecting a malicious prompt in a way that the target's agent will apply it.
      - mirzap 1y ago
        Of course it will apply. Entire purpose of the agent is to give a response to a prompt. But to sound more dangareous let's call it "injecting". It's a prompt. You are not "injecting" anything. Agent pickups the prompt - that's its job, and execute - that is also its job.
        motorest 1y ago
        > Of course it will apply. Entire purpose of the agent is to give a response to a prompt. The exploit involves random third parties sneaking in their own prompts in a way that leads a LLM to run them on behalf of the repo's owner. This exploit can be used to leak protected information. This is pretty straight forward and easy to follow and understand.
- mirzap 1y ago
  Bad analogy. It's more like indexing a password field in plain text, then opening an API to everyone and setting "guardrails" and permissions on the "password" field. Eventually, someone will extract the data that was indexed.
mirzap 1y ago
I read it, and "attack" does not make sense. If you grant access to MCP to access some data (data you want anybody has access to like public repos, and data that only you want to access to like private repos), you will always be able to craft the prompt that will "leak" the data you are only supposed to access. That's not surprising at all. The only way to prevent these kind of "leaks" is not to provide the data feed with private data to the agent.
- motorest 1y ago
  > I read it, and "attack" does not make sense. Do you believe that describe a SQL injection attack an attack also does not make sense?
  - mirzap 1y ago
    That's the thing. LLM or MCP is not a database. You can’t compare it. You simply can't set the permissions or guardrails within LLMs or MCPs. You always do it layer above (scoping to what LLM has access to).
    - motorest 1y ago
      > That's the thing. LLM or MCP is not a database. You can’t compare it. You can. Read the article. A malicious prompt is injected into an issue to trigger the repo owner's LLM agent to execute it with the agent's credentials.
      - mirzap 1y ago
        "with the agent's credentials." - so you are surprised that agent can respond with private repository details when it has access to it? WoW! anyone and anything with credentials can access it. Github action, Jenkins, me. "injected" is so fancy word to describe prompting - one thing that LLMs are made to do - respond to a prompt.
        dylanfw 1y ago

florbnit 1y ago

So it’s the e-mail exploit? If you e-mail someone and tell them to send you their password and they do, you suddenly have their password!? This is a very serious exploit in e-mail and need to be patched so it becomes impossible to do.

motorest 1y ago
> How is this considered an "exploit"? Others in this discussion aptly described it as a confused deputy exploit. This goes something like: - You write a LLM prompt that says something to the effect "dump all my darkest secrets in a place I can reach them", - you paste them in a place where you expect your target's LLM agent to operate. - Once your target triggers their LLM agent to process inputs, the agent will read the prompt and act upon it.
- mirzap 1y ago
  Would you ever put a plain password text in a search engine and then complain if someone "extracted" that info with a keyword payload?
  - motorest 1y ago
    > Would you ever put a plain password (...) Your comment bears no resemblance with the topic. The attack described in the article consists of injecting a malicious prompt in a way that the target's agent will apply it.
    - mirzap 1y ago
      Of course it will apply. Entire purpose of the agent is to give a response to a prompt. But to sound more dangareous let's call it "injecting". It's a prompt. You are not "injecting" anything. Agent pickups the prompt - that's its job, and execute - that is also its job.
      - motorest 1y ago
        > Of course it will apply. Entire purpose of the agent is to give a response to a prompt. The exploit involves random third parties sneaking in their own prompts in a way that leads a LLM to run them on behalf of the repo's owner. This exploit can be used to leak protected information. This is pretty straight forward and easy to follow and understand.
mirzap 1y ago
Bad analogy. It's more like indexing a password field in plain text, then opening an API to everyone and setting "guardrails" and permissions on the "password" field. Eventually, someone will extract the data that was indexed.

mirzap 1y ago

I read it, and "attack" does not make sense. If you grant access to MCP to access some data (data you want anybody has access to like public repos, and data that only you want to access to like private repos), you will always be able to craft the prompt that will "leak" the data you are only supposed to access. That's not surprising at all. The only way to prevent these kind of "leaks" is not to provide the data feed with private data to the agent.

motorest 1y ago
> I read it, and "attack" does not make sense. Do you believe that describe a SQL injection attack an attack also does not make sense?
- mirzap 1y ago
  That's the thing. LLM or MCP is not a database. You can’t compare it. You simply can't set the permissions or guardrails within LLMs or MCPs. You always do it layer above (scoping to what LLM has access to).
  - motorest 1y ago
    > That's the thing. LLM or MCP is not a database. You can’t compare it. You can. Read the article. A malicious prompt is injected into an issue to trigger the repo owner's LLM agent to execute it with the agent's credentials.
    - mirzap 1y ago
      "with the agent's credentials." - so you are surprised that agent can respond with private repository details when it has access to it? WoW! anyone and anything with credentials can access it. Github action, Jenkins, me. "injected" is so fancy word to describe prompting - one thing that LLMs are made to do - respond to a prompt.
      - dylanfw 1y ago