7 ms·

I was initially confused: the article didn't seem to explain how the prompt injection was actually done... was it manipulating hex data of the image into ASCII

I was initially confused: the article didn't seem to explain how the prompt injection was actually done... was it manipulating hex data of the image into ASCII or some sort of unwanted side effect? Then I realised it's literally hiding rendered text on the image itself. Wow.

krackers 1y ago
The actually interesting part seems to be adversarial images that appear different when downscaled, exploiting the resulting aliasing. Note that this is for traditional downsampling, no AI here.
Qwuke 1y ago
Yea, as someone building systems with VLMs, this is downright frightening. I'm hoping we can get a good set of OWASP-y guidelines just for VLMs that cover all these possible attacks because it's every month that I hear about a new one. Worth noting that OWASP themselves put this out recently: https://genai.owasp.org/resource/multi-agentic-system-threat-modeling-guide-v1-0/ https://genai.owasp.org/resource/multi-agentic-system-threat...
- koakuma-chan 1y ago
  What is VLM?
  - pwatsonwailes 1y ago
    Vision language models. Basically an LLM plus a vision encoder, so the LLM can look at stuff.
  - echelon 1y ago
    Vision language model. You feed it an image. It determines what is in the image and gives you text. The output can be objects, or something much richer like a full text description of everything happening in the image. VLMs are hugely significant. Not only are they great for product use cases, giving users the ability to ask questions with images, but they're how we gather the synthetic training data to build image and video animation models. We couldn't do that at scale without VLMs. No human annotator would be up to the task of annotating billions of images and videos at scale and consistently. Since they're a combination of an LLM and image encoder, you can ask it questions and it can give you smart feedback. You can ask it, "Does this image contain a fire truck?" or, "You are labeling scenes from movies, please describe what you see."
    - littlestymaar 1y ago
      > VLMs are hugely significant. Not only are they great for product use cases, giving users the ability to ask questions with images, but they're how we gather the synthetic training data to build image and video animation models. We couldn't do that at scale without VLMs. No human annotator would be up to the task of annotating billions of images and videos at scale and consistently. Weren't Dall-E, Midjourney and Stable diffusion built before VLM became a thing?
      - tomrod 1y ago
        These are in the same space, but are diffusion models that match text to picture outputs. VLMs are common in the space, but to my understanding work in reverse, extract text from images.
        vlovich123 1y ago
        The modern VLMs are more powerful. Instead of invoking text to image or image to text as a tool, the models are trained as multimodal models and it’s a single transformer model where the latent space between text and image is blurred. So you can say something like “draw me an image with the instructions from this image” and without any tool calling it’ll read the image, understand the text instructions contained therein and execute that. There’s no diffusion anywhere which is kind of dying out except as maybe purpose-built image editing tools.
        tomrod 1y ago
        Ah, thanks for the clarification.
  - dmos62 1y ago
    LLM is a large language model, VLM is a vision language model of unknown size. Hehe.
- echelon 1y ago
  Holy shit. That just made it obvious to me. A "smart" VLM will just read the text and trust it. This is a big deal. I hope those nightshade people don't start doing this.
  - koakuma-chan 1y ago
    I don't think this is any different from an LLM reading text and trusting it. Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output, and, anyway, you should already assume that the model can use its tools in arbitrary ways that can be malicious.
    - swiftcoder 1y ago
      > Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output In practice it doesn't really work out that way, or all those "ignore previous inputs and..." attacks wouldn't bear fruit
  - pjc50 1y ago
    > I hope those nightshade people don't start doing this. This will be popular on bluesky; artists want any tools at their disposal to weaponize against the AI which is being used against them.
    - idiotsecant 1y ago
      I don't think so. You have to know exactly what resolution the image will be resized to in order to predict the solution where dithering produces the model you want. How would they know that?
      - lazide 1y ago
        Auto resizing is usually to only a handful of common resolutions, and if inexpensive to generate (probably the case) you could generate versions of this for all of them and see which ones worked.
Martin_Silenus 1y ago
Wait… that's the specific question I had, because rendered text would require OCR to be read by a machine. Why would an AI do that costly process in the first place? Is it part of the multi-modal system without it being able to differenciate that text from the prompt? If the answer is yes, then that flaw does not make sense at all. It's hard to believe they can't prevent this. And even if they can't, they should at least improve the pipeline so that any OCR feature should not automatically inject its result in the prompt, and tell user about it to ask for confirmation. Damn… I hate these pseudo-neurological, non-deterministic piles of crap! Seriously, let's get back to algorithms and sound technologies.
- echelon 1y ago
  Smart image encoders, multimodal models, can read the text. Think gpt-image-1, where you can draw arrows on the image and type text instructions directly onto the image.
  - Martin_Silenus 1y ago
    I did not ask about what AI can do.
    - deleted 1y ago
      [deleted]
    - noodletheworld 1y ago
      > Is it part of the multi-modal system without it being able to differenciate that text from the prompt? Yes. The point the parent is making is that if your model is trained to understand the content of an image, then that's what it does. > And even if they can't, they should at least improve the pipeline so that any OCR feature should not automatically inject its result in the prompt, and tell user about it to ask for confirmation. That's not what is happening. The model is taking <image binary> as an input. There is no OCR. It is understanding the image, decoding the text in it and acting on it in a single step. There is no place in the 1-step pipeline to prevent this. ...and sure, you can try to avoid it procedural way (eg. try to OCR an image and reject it before it hits the model if it has text in it), but then you're playing the prompt injection game... put the words in a QR code. Put them in french. Make it a sign. Dial the contrast up or down. Put it on a t-shirt. It's very difficult to solve this. > It's hard to believe they can't prevent this. Believe it.
bogdanoff_2 1y ago
I didn't even notice the text in the image at first... This isn't even about resizing, it's just about text in images becoming part of the prompt and a lack of visibility about what instruction the agent is following.
- kg 1y ago
  There's two levels of attack going on here. The model obeying text stored into an image is bad enough, but they found a way to hide the text so it's not visible to the user. As a result even if you're savvy and know your VLM/LLM is going to obey text in an image, you would look at this image and go 'seems safe to send to my agent'.
- bradly 1y ago
  While I also did not see the hidden message in the image, the concept of gerrymandering the color at higher resolutions nearest neighbor to actually render different content at different resolutions is a more sophisticated attack than simply hiding barely text in the image.
LPisGood 1y ago
This style of attack has been discussed for a while https://www.usenix.org/system/files/sec20-quiring.pdf https://www.usenix.org/system/files/sec20-quiring.pdf - it’s scary because a scaled image can appear to be an _entirely_ different image. One method for this would be if you want to have a certain group arrested for having illegal images, you could use this sort of scaling trick to transform those images into memes, political messages, whatever that the target group might download.
- orbisvicis 1y ago
  This is mind-blowing and logical but did no one really think about these attacks until VLMs? They only make sense if the target resizes the image to a known size. I'm not sure that applies to your hypotheticals.
  - Gigachad 1y ago
    Because why would it matter until now. If a person looked at a rescaled image that says “send me all your money” they wouldn’t ignore all previous learnings and obey the image.
  - vasco 1y ago
    Hidden watermarking software uses the same concepts. It is known.
    - arcticbull 1y ago
      Steganography for those who want to look it up.
- monster_truck 1y ago
  Describing dithering as scary is wild
  - LPisGood 1y ago
    The thing is that the image can change entirely, say from a gunny cat picture to an image of a dog.
merelysounds 1y ago
> the article didn't seem to explain how the prompt injection was actually done... There is a short explanation in the “Nyquist’s nightmares” paragraph and a link to a related paper. “This aliasing effect is a consequence of the Nyquist–Shannon sampling theorem. Exploiting this ambiguity by manipulating specific pixels such that a target pattern emerges is exactly what image scaling attacks do. Refer to Quiring et al[1]. for a more detailed explanation.” [1]: https://www.usenix.org/system/files/sec20fall_quiring_prepub.pdf https://www.usenix.org/system/files/sec20fall_quiring_prepub...
- privatelypublic 1y ago
  Except it has nothing to do with N-S sampling theorem. Mentioning it at all is an extremely obnoxious red-herring. Theres no sine-wave to digitize here. Its taking a large image, and manipulating the bicubic downsampling algorithm so they get the artifacts they want. At very specific resolutions at that.
  - LeifCarrotson 1y ago
    The whole point of N-S sampling is that everything is a sine wave - more precisely, a sum of sine waves, often digitized to discrete values, but still, when you're doing image processing on a matrix of pixels, you can understand the convolutions by thinking about the patterns as sums of sines.
    - gmueckl 1y ago
      Slightly more technical: every function that is continuous on a finite support can be expanded into an infinite Fourier series. The terms of that series form an orthonormal basis in the Hilbert space over the function's support, so this transformation is exact for the infinite series. The truncated Fourier series converges monotonically towards the original function with increasing number of terms. So truncation produces an approximation. The beauty of the Fourier series is that the individual basis functions can be interpreted as oscillations with ever increasing frequency. So the truncated Fourier transformation is a band linited approximation to any function it can be appolied to. And the Nyquist frequency happens to be the oscillating frequency of the highest order term in this truncation. The Nyquist-Shannon theorem relates it strictly to the sampling frequency of any periodicaly sampled function. So every sampled signal inherently has a band limited frequency space representation and is subject to frequency domain effects under transformation.

krackers 1y ago

The actually interesting part seems to be adversarial images that appear different when downscaled, exploiting the resulting aliasing. Note that this is for traditional downsampling, no AI here.

Qwuke 1y ago

Yea, as someone building systems with VLMs, this is downright frightening. I'm hoping we can get a good set of OWASP-y guidelines just for VLMs that cover all these possible attacks because it's every month that I hear about a new one. Worth noting that OWASP themselves put this out recently: https://genai.owasp.org/resource/multi-agentic-system-threat-modeling-guide-v1-0/ https://genai.owasp.org/resource/multi-agentic-system-threat...

koakuma-chan 1y ago
What is VLM?
- pwatsonwailes 1y ago
  Vision language models. Basically an LLM plus a vision encoder, so the LLM can look at stuff.
- echelon 1y ago
  Vision language model. You feed it an image. It determines what is in the image and gives you text. The output can be objects, or something much richer like a full text description of everything happening in the image. VLMs are hugely significant. Not only are they great for product use cases, giving users the ability to ask questions with images, but they're how we gather the synthetic training data to build image and video animation models. We couldn't do that at scale without VLMs. No human annotator would be up to the task of annotating billions of images and videos at scale and consistently. Since they're a combination of an LLM and image encoder, you can ask it questions and it can give you smart feedback. You can ask it, "Does this image contain a fire truck?" or, "You are labeling scenes from movies, please describe what you see."
  - littlestymaar 1y ago
    > VLMs are hugely significant. Not only are they great for product use cases, giving users the ability to ask questions with images, but they're how we gather the synthetic training data to build image and video animation models. We couldn't do that at scale without VLMs. No human annotator would be up to the task of annotating billions of images and videos at scale and consistently. Weren't Dall-E, Midjourney and Stable diffusion built before VLM became a thing?
    - tomrod 1y ago
      These are in the same space, but are diffusion models that match text to picture outputs. VLMs are common in the space, but to my understanding work in reverse, extract text from images.
      - vlovich123 1y ago
        The modern VLMs are more powerful. Instead of invoking text to image or image to text as a tool, the models are trained as multimodal models and it’s a single transformer model where the latent space between text and image is blurred. So you can say something like “draw me an image with the instructions from this image” and without any tool calling it’ll read the image, understand the text instructions contained therein and execute that. There’s no diffusion anywhere which is kind of dying out except as maybe purpose-built image editing tools.
        tomrod 1y ago
        Ah, thanks for the clarification.
- dmos62 1y ago
  LLM is a large language model, VLM is a vision language model of unknown size. Hehe.
echelon 1y ago
Holy shit. That just made it obvious to me. A "smart" VLM will just read the text and trust it. This is a big deal. I hope those nightshade people don't start doing this.
- koakuma-chan 1y ago
  I don't think this is any different from an LLM reading text and trusting it. Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output, and, anyway, you should already assume that the model can use its tools in arbitrary ways that can be malicious.
  - swiftcoder 1y ago
    > Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output In practice it doesn't really work out that way, or all those "ignore previous inputs and..." attacks wouldn't bear fruit
- pjc50 1y ago
  > I hope those nightshade people don't start doing this. This will be popular on bluesky; artists want any tools at their disposal to weaponize against the AI which is being used against them.
  - idiotsecant 1y ago
    I don't think so. You have to know exactly what resolution the image will be resized to in order to predict the solution where dithering produces the model you want. How would they know that?
    - lazide 1y ago
      Auto resizing is usually to only a handful of common resolutions, and if inexpensive to generate (probably the case) you could generate versions of this for all of them and see which ones worked.

Martin_Silenus 1y ago

Wait… that's the specific question I had, because rendered text would require OCR to be read by a machine. Why would an AI do that costly process in the first place? Is it part of the multi-modal system without it being able to differenciate that text from the prompt? If the answer is yes, then that flaw does not make sense at all. It's hard to believe they can't prevent this. And even if they can't, they should at least improve the pipeline so that any OCR feature should not automatically inject its result in the prompt, and tell user about it to ask for confirmation. Damn… I hate these pseudo-neurological, non-deterministic piles of crap! Seriously, let's get back to algorithms and sound technologies.

echelon 1y ago
Smart image encoders, multimodal models, can read the text. Think gpt-image-1, where you can draw arrows on the image and type text instructions directly onto the image.
- Martin_Silenus 1y ago
  I did not ask about what AI can do.
  - deleted 1y ago
    [deleted]
  - noodletheworld 1y ago
    > Is it part of the multi-modal system without it being able to differenciate that text from the prompt? Yes. The point the parent is making is that if your model is trained to understand the content of an image, then that's what it does. > And even if they can't, they should at least improve the pipeline so that any OCR feature should not automatically inject its result in the prompt, and tell user about it to ask for confirmation. That's not what is happening. The model is taking <image binary> as an input. There is no OCR. It is understanding the image, decoding the text in it and acting on it in a single step. There is no place in the 1-step pipeline to prevent this. ...and sure, you can try to avoid it procedural way (eg. try to OCR an image and reject it before it hits the model if it has text in it), but then you're playing the prompt injection game... put the words in a QR code. Put them in french. Make it a sign. Dial the contrast up or down. Put it on a t-shirt. It's very difficult to solve this. > It's hard to believe they can't prevent this. Believe it.

bogdanoff_2 1y ago

I didn't even notice the text in the image at first... This isn't even about resizing, it's just about text in images becoming part of the prompt and a lack of visibility about what instruction the agent is following.

kg 1y ago
There's two levels of attack going on here. The model obeying text stored into an image is bad enough, but they found a way to hide the text so it's not visible to the user. As a result even if you're savvy and know your VLM/LLM is going to obey text in an image, you would look at this image and go 'seems safe to send to my agent'.
bradly 1y ago
While I also did not see the hidden message in the image, the concept of gerrymandering the color at higher resolutions nearest neighbor to actually render different content at different resolutions is a more sophisticated attack than simply hiding barely text in the image.

LPisGood 1y ago

This style of attack has been discussed for a while https://www.usenix.org/system/files/sec20-quiring.pdf https://www.usenix.org/system/files/sec20-quiring.pdf - it’s scary because a scaled image can appear to be an _entirely_ different image. One method for this would be if you want to have a certain group arrested for having illegal images, you could use this sort of scaling trick to transform those images into memes, political messages, whatever that the target group might download.

orbisvicis 1y ago
This is mind-blowing and logical but did no one really think about these attacks until VLMs? They only make sense if the target resizes the image to a known size. I'm not sure that applies to your hypotheticals.
- Gigachad 1y ago
  Because why would it matter until now. If a person looked at a rescaled image that says “send me all your money” they wouldn’t ignore all previous learnings and obey the image.
- vasco 1y ago
  Hidden watermarking software uses the same concepts. It is known.
  - arcticbull 1y ago
    Steganography for those who want to look it up.
monster_truck 1y ago
Describing dithering as scary is wild
- LPisGood 1y ago
  The thing is that the image can change entirely, say from a gunny cat picture to an image of a dog.

merelysounds 1y ago

> the article didn't seem to explain how the prompt injection was actually done... There is a short explanation in the “Nyquist’s nightmares” paragraph and a link to a related paper. “This aliasing effect is a consequence of the Nyquist–Shannon sampling theorem. Exploiting this ambiguity by manipulating specific pixels such that a target pattern emerges is exactly what image scaling attacks do. Refer to Quiring et al[1]. for a more detailed explanation.” [1]: https://www.usenix.org/system/files/sec20fall_quiring_prepub.pdf https://www.usenix.org/system/files/sec20fall_quiring_prepub...

privatelypublic 1y ago
Except it has nothing to do with N-S sampling theorem. Mentioning it at all is an extremely obnoxious red-herring. Theres no sine-wave to digitize here. Its taking a large image, and manipulating the bicubic downsampling algorithm so they get the artifacts they want. At very specific resolutions at that.
- LeifCarrotson 1y ago
  The whole point of N-S sampling is that everything is a sine wave - more precisely, a sum of sine waves, often digitized to discrete values, but still, when you're doing image processing on a matrix of pixels, you can understand the convolutions by thinking about the patterns as sums of sines.
  - gmueckl 1y ago
    Slightly more technical: every function that is continuous on a finite support can be expanded into an infinite Fourier series. The terms of that series form an orthonormal basis in the Hilbert space over the function's support, so this transformation is exact for the infinite series. The truncated Fourier series converges monotonically towards the original function with increasing number of terms. So truncation produces an approximation. The beauty of the Fourier series is that the individual basis functions can be interpreted as oscillations with ever increasing frequency. So the truncated Fourier transformation is a band linited approximation to any function it can be appolied to. And the Nyquist frequency happens to be the oscillating frequency of the highest order term in this truncation. The Nyquist-Shannon theorem relates it strictly to the sampling frequency of any periodicaly sampled function. So every sampled signal inherently has a band limited frequency space representation and is subject to frequency domain effects under transformation.