3 ms·
Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stol
by simonw 1y ago
Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack.
There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/ https://simonwillison.net/2025/Jun/11/echoleak/
My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
- sandspar 1y agoSooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.
- LPisGood 1y agoI mean this was a threat before - google, ISPs, and adult sites that require payment have been places one could attack to make this for decades.
- potato3732842 1y agoYou can already do this with ad network tech.
- andy99 1y agoI'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild. I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
- simonw 1y agoAn attack where somebody emails the VP of marketing of a company and gets automatically leaked their latest sales figures feels pretty impactful to me. Or emails the CEO and gets automatically forwarded all of their password reset messages.
- simonw 1y agoOut of interest, when you said "mostly overblown" were you thinking of prompt injection as "people inject bad prompts into an LLM" (I'd call that "jailbreaking") or were you thinking of the thing where an attacker sneaks bad instructions into the model against the wishes of the model's actual user, eg by hiding instructions on a web page that gets summarized or sending them to the victim in an email?
- deleted 1y ago[deleted]