4 ms·
Microsoft 365 Copilot – Arbitrary Data Exfiltration via Mermaid Diagrams
https://web.archive.org/web/20251023095538/https://www.adamlogue.com/microsoft-365-copilot-arbitrary-data-exfiltration-via-mermaid-diagrams-fixed/ https://web.archive.org/web/20251023095538/https://www.adaml...
- simonw 1y agoThat site just gave me a 503 but here's the Internet Archive copy: https://web.archive.org/web/20251023095538/https://www.adamlogue.com/microsoft-365-copilot-arbitrary-data-exfiltration-via-mermaid-diagrams-fixed/ https://web.archive.org/web/20251023095538/https://www.adaml... This isn't the first Mermaid prompt injection exfiltration we've seen - here's one from August that was reported by Johann Rehberger against Cursor (and fixed by them): https://embracethered.com/blog/posts/2025/cursor-data-exfiltration-with-mermaid/ https://embracethered.com/blog/posts/2025/cursor-data-exfilt... That's mentioned in the linked post. Looks like that attack was different - Cursor's Mermaid implementation could render external images, but Copilot's doesn't let you do that so you need to trick users with a fake Login button that activates a hyperlink instead.
- luke-stanley 1y agoThe Lethal Trifecta strikes again! Mermaid seems like a bit of a side issue, presumably there are lots of ways data might leak out. It could have just been a normal link. They should probably look further into the underlying issue: unrelated instruction following. Thanks for the archive link and the very useful term BTW! I also got 503 when trying to visit.
- simonw 1y agoI think they're doing this the right way. You can't fix unrelated instruction following with current generation LLMs, so given that the only leg you can remove from the trifecta is mechanisms for exfiltrating the data. The first AI lab to solve unrelated instruction following is going to have SUCH a huge impact.
- hshdhdhehd 1y agoNot even humans can do it perfectly (hence social engineering)
- luke-stanley 1y agoHumans would not do what was reported here!
- MattPalmer1086 1y agoNo, they are doing this the only possible way that doesn't massively restrict it being useful at all. That doesn't make it the right way. A fundamental vulnerability to prompt injection means pretty much any output can be dangerous, and they have to expose it to largely untrusted input to be useful at all. Even limiting output to ASCII text only is probably not entirely safe. The right way at this point would be to not use AI.
- luke-stanley 1y agoSummarisation models that do not follow instructions already exist! Fixing exfiltration is good, low hanging fruit. But for a summarisation task, whole classes of typical instruction following behaviour are totally off target!
- simonw 1y agoWhich models do you recommend for that these days? Has the field moved on much since the pre-LLM-era models like T5?
- binarymax 1y ago> MSRC bounty team determined that M365 Copilot was out-of-scope for bounty and therefore not eligible for a reward. What a shame. There’s probably LOTS of vulns in copilot. This just discourages researchers and responsible disclosure, likely leaving copilot very insecure in the long run.
- CaptainOfCoit 1y ago> There’s probably LOTS of vulns in copilot Probably exactly why they "determined" it to be out of scope :)
- candiddevmike 1y agoIt's irresponsible for any company to be using copilot with MS having this bug bounty attitude, IMO. Would be curious what other products are out of bounds so I know not to use them...
- kenjackson 1y agoIs there any company that has bug bounties on all their products?
- oowahahahah 1y ago[dead]
- AppleBananaPie 1y agoI'd assume the app / technology Microsoft is pushing over all else is more worth a bug bounty than say Visio so maybe more accurate to ask are there any major companies with their new key product that don't have bug bounties? Happy to be wrong and put my foot in my mouth though I've misunderstood folks before :)
- fingerlocks 1y agoThis is about the M365 Copilot, which is the enterprise version. The normal consumer version of Copilot shares nothing in common except for the name, at least that’s what I’ve been told. Different architecture, team, back end, etc. And side by side, the enterprise version is much crapper of the two.
- a-dub 1y ago" ... BUT most importantly, ... " i love the use of all capitals for emphasis for important instructions in the malicious prompt. it's almost like an enthusiastic leader of a criminal gang explaining the plot in a dingey diner the night before as the rain pours outside.
- Nextgrid 1y agoIt’s both interesting to see all the creative ways people find to exploit LLM-based systems, but also disappointing that to this day designers of these systems don’t want to accept that LLMs are inherently vulnerable to prompt injection and short of significant breakthroughs in AI interpretability will remain hopelessly broken regardless of ad-hoc “mitigations” they implement.
- chasing0entropy 1y agoI am of the opinion LLMs are cognitive and task capability equivalent of a 5 year old. Actually that might be a harsh judgement since a child will succeed with practice.
- vuln 1y agoSo does a monkey or a dolphin, what's your point?
- singingfish 1y agoaka LLMs can not learn from experience - this is a fundamental limitation. c.f - individuals with Korsakov's syndrome - who also confabulate in a similar manner.
- narrator 1y agoPrompt Injection is an interesting difference between human consciousness and machine "consciousness", or what people try and liken to it. A human can easily tell when information is coming from his memory or internal thoughts and when it is coming from a possibly less reliable outside source. Gaslighting is essentially an attempted prompt injection and is considered psychological abuse. Interestingly, people complain about AI gaslighting them and AI doesn't seem to think that's a problem.
- lazyasciiart 1y agoIsn’t that what marketing is?
- chasing0entropy 1y agoZing. I was about to remark on the fascinating paradigm of AI emotionally abusing humans without consequence but now the rabbit hole has deepened to considering what level of advertising could also be justified as emotionally abusive and how many layers of legislative protection may already exist to pre-empt the argument.
- grantcas 1y ago[dead]
- mrasong 1y agoWe're going to see a new kind of hacker — prompt-injection attacks.
- MattPalmer1086 1y agoWe already have.