7 ms·
Multi-modal prompt injection image attacks against GPT-4V
- Roritharr 3y agoI love simonw content. It's like a day after I saw something interesting on X wrt AI, I can trust that there will be an article a day later by simonw where he tried the things I would have wanted to try and explains them nicely. Thanks for that!
- sureglymop 3y agoSame sentiment. Its like he reads hacker news or similar sources and is generally on top of tech news and then has the curiosity to look more into it himself and do practical experiments. I try to do the same which is why I love this content.
- jgalt212 3y agoI feel differently. I think Simon's wasting his talents on an evolutionary dead end. If transformers and LLMs held the answer, self-driving cars would be getting better.
- simonw 3y agoAs a daily user of LLMs for over a year, I'm confident that they're not a waste of my time. Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using. I expect we could still spend years figuring out new capabilities in the models we already have access to today.
- wunderwuzzi23 3y agoI'm very glad that you are focusing and helping educate the industry (both from pure tech but also security aspects). And agreed that as a daily user of LLMs myself, the potential (and practical use cases already) are enormous. E.g. Writing code and rapid prototyping will never be the same for me. By the way the last official answer I got from OpenAI regarding the markdown issue was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution. My current takeaway is that with LLMs, the assumption always needs to be that the LLM is capable of outputting any content, and the client needs to safely (and securely) handle whatever is returned. Btw. I show cased these kind of image based prompt injections in July (with Bard and Bing Chat, which presumably already used OpenAI's tech), and the example I created (the robot slipping on a banana) was shown during Blackhat in Las Vegas this year. Tweets: https://twitter.com/wunderwuzzi23/status/1681520761146834946 https://twitter.com/wunderwuzzi23/status/1681520761146834946
- simonw 3y agoJust added your tweet there to my Twitter thread about this: https://twitter.com/simonw/status/1713344669181178088 https://twitter.com/simonw/status/1713344669181178088
- danShumway 3y agoSimon is bullish about LLMs, but his writing has been incredibly valuable regardless of which side of the LLM fence you're on. There are a limited number of authors describing injection attacks in accessible, detailed, accurate ways, and Simon has done a lot to raise awareness about these attacks. In addition, the fact that Simon is very bullish about LLMs makes his writing on LLM attacks more effective, because a decent chunk of people warning about LLM attacks are also skeptical about the long-term viability of the technology or think that it's over-hyped, so LLM fans tend to to dismiss them as being "anti-AI" and say that they're playing up the dangers. Nobody can make that claim about Simon. For whatever reason Simon's writing has consistently attracted a lot more attention than other researcher releases about LLM attacks; and what has he done with that popularity and outsized attention? He's raised awareness about the issue and pointed back to some of that research! I just don't think the criticism in this case is warranted, I think Simon's writing here is worthwhile regardless of how anyone feels about the potential of LLMs. Whether they're the future or not, these vulnerabilities exist today in products that are being rolled out today. That's worth taking about if LLMs aren't the future, but it's even more worth talking about if LLMs are the future.
- sebzim4500 3y agoWhat do LLMs have to do with self-driving cars? Also they are getting better, just slowly.
- famouswaffles 3y agoGuess he's expecting something like this to happen faster? https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-...
- jgalt212 3y agoMy analogy is that you cannot trust the outputs of an LLM or self-driving car.
- jug 3y agoWell, self-driving cars are already in production and use. They may not be amazing just yet but they have already passed the greatest hurdle there is.
- ewngzen 3y agoself-driving cars already in production are not explainable, at least with llm in it, we know what the network is thinking, and thus are possible to figure out what went wrong
- htrp 3y agoVision is going to be significantly harder to filter for due to the higher dimensional nature of the inputs.
- mk_stjames 3y agoI'm still a bit surprised at the visual acuity of GPT-4V. The spatial resolution is far finer than I thought we were at right now- for instance, if you had asked me what I thought a model would return as far as a description of the pumpkin weigh scene goes, I'd have said "crowd of people, cameras, chart, house" etc- you know, like YOLOv8 level of identification. But it gets down to picking up numbers and letters that are tiny, gets obscured text like half moon bay (is it using a combination of the scene letters and knowledge in its language model of a pumpkin contest in HMB and synthesizing that knowledge into recognizing the location!!!!???). It's damn detailed.
- simonw 3y agoYeah, I think it's spectacular - GPT4-V is so much better then I expected it to be.
- beebmam 3y agoIt's fun to ask it to try to draw ascii art based on an image. It barely draws anything resembling the original image, but usually gets the concepts in the original image accurately.
- jcims 3y agoI’m super curious about the data structure that’s being extracted from the image and fed to GPT. I asked it to estimate the size of a figurine based on a ruler in the image, and it nailed it. https://imgur.com/a/gsiID4v https://imgur.com/a/gsiID4v At first I just thought it was able taking an array of objects identified in the photo with bounding box coordinates and then just figuring out what the answer is. But then i asked it to describe Worf’s hands in this image and it did a great job. https://imgur.com/a/orpDszj https://imgur.com/a/orpDszj I’m super curious. I’ve done some basic attempts to get it to spit out the actual raw data but it won’t. I’m sure folks that are good at that could.
- simonw 3y agoI don't think that's the right way to think about it. It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing. Instead, the model itself has been trained on both text and images at the same time. So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt. I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?
- magicalhippo 3y agoNot my field but I'd assume they had encoded tokens extracted from an image differently than prompt tokens, so they wouldn't get interpreted as a command. Why aren't they doing this? Or are they, it's just failing?
- simonw 3y agoI don't think anyone has figured out how to implement this kind of LLM system in that way. If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! It's been 13 months since we started discussing this vulnerability and I've not seen any evidence of anyone cracking this yet.
- aleph_minus_one 3y ago> If you can crack differentiating between "command" tokens and other input tokens, you've cracked prompt injection! In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. For example, the user would mark his prompt (that he thoroughly thought through) as "trusted", but the image which provides additional information, or is the object to run the prompt on, as "untrusted". This is related to established concepts in IT security: - W^X: memory pages are either writable or executable, but not both (of course the term "W^X" is a little bit confusing since if W=0 and X=0, W^X is 0, but is an allowed configuration) - corresponding to untrusted or trusted parts of the prompt - sandboxing of the virtual machine in browsers that execute untrusted data from the web (untrusted part)
- simonw 3y ago> the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way. That's exactly the solution I've been hoping for, but seeing as no-one has even come close to getting that to work yet my current intuition is that it's REALLY hard. Maybe even impossible with current LLM architectures?
- danShumway 3y ago> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers. Everyone always is because why on earth wouldn't this be blocked? That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabled in places like Bing chat" and OpenAI in particular seems to have consistently decided that it's a core part of the chat capabilities for some reason -- it's on of the reasons why I say that OpenAI does not take security seriously. It really should not be a debate whether or not remote images should be blocked from markdown. Your email client blocks remote images. Most forums either block remote images or proxy them. There is no reason at all for ChatGPT to have that the ability to display them. Every company makes security mistakes, that's not a problem; it's when a company doubles down on security mistakes that their attitude about security becomes apparent.
- spdustin 3y agoI use it all the time to embed graphviz diagrams by instructing it to url-encode them and include it in the query string to a rendering endpoint. It spooked me enough that I added proxy layer to verify that it’s not rendering what appears to be a chat or PII. The world isn’t ready for LLMs unleashed like this. Any plugin could do the same thing, and exfiltrate any part of the chat context with very little instruction engineering to bypass what things guardrails exist on the plugins side.
- wunderwuzzi23 3y agoThanks for the shout out, the last answer I got from OpenAI in that regards was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution. My conclusion is that with LLMs, the assumption always needs to be that the LLM might be capable of outputting any content, and the client needs to safely handle whatever is returned.
- Terretta 3y agoThe leak is not the remote image. The leak is the URL request.
- gitfan86 3y agoThis is the biggest problem in cyber security. You can tell someone to NEVER give out their TFA code over the phone and they will still do it when prompted by a phone call from their boss.
- Terretta 3y agoI'm going into a meeting and need to give some iTunes Gift Cards as an award. Can you grab me a few from an Apple store? -- from your new boss, according to LinkedIn data
- saberience 3y agoThis post broke my brain a little as it goes totally against my mental model of how an LLM is processing images. That is, I don't know why it is processing the content of the image, and then running that through something that then "executes" the text of the image. So it has some pipeline like -> LLM/OCR Image to Text -> Text to Command Processing. Why is the text description of the image being processed at all and not simply just being output? As far as I know, we've had engines for detecting sentiment, objects, etc from images for a long time (way before ChatGPT), and they didn't have issues where they would execute a URL found in an image, they would just output the characters... Can someone who understands this better explain wth is going on here?
- Michelangelo11 3y agoInteresting that you say that, because this lines up exactly with my mental models of LLM image processing and LLM functioning in general. One of the defining features of LLMs is that, unlike in regular programming, there's no separation between data and instructions -- they're one and the same -- and that's what we're seeing here. Also, LLMs tend to focus more heavily on the instructions they see last (because of limitations in the attention mechanism, I think? But IIRC nobody has come up with a full explanation yet), and that's what we're seeing here too.
- saberience 3y agoSo the original release of ChatGPT would call a URL you entered in the chat? I don't recall it acting that way. In the linked post ChatGPT literally goes to a URL from an image... i.e. it calls something like curl with that URL. This definitely seems like something adjacent to an LLM.
- simonw 3y agoChatGPT didn't grow the ability to visit URLs until they introduced ChatGPT Browse - though it frequently hallucinated that it could: https://simonwillison.net/2023/Mar/10/chatgpt-internet-access/ https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... The exfiltration demo in the blog post doesn't show ChatGPT fetching a URL: it shows it outputting an image tag with an src= pointing at an external URL, which the user's browser then attempts to load.
- nullc 3y agoIt sure seems that a lot of effort goes in to preventing the LLM from obeying the user while actual security issues-- like allowing CSRF like attacks from the interface-- go ignored. If the LLM embedding content from arbitrary sources is really such a critical feature it could at least be behind a click that shows you the URL.