6 ms·

This seems to introduce levels of artifacts that many artists would find unacceptable: https://twitter.com/sini4ka111/status/1748378223291912567 https://twitter

by 542458 3y ago

This seems to introduce levels of artifacts that many artists would find unacceptable: https://twitter.com/sini4ka111/status/1748378223291912567 https://twitter.com/sini4ka111/status/1748378223291912567 The rumblings I'm hearing are that this a) barely works with last-gen training processes b) does not work at all with more modern training processes (GPT-4V, LLaVA, even BLIP2 labelling [1]) and c) would not be especially challenging to mitigate against even should it become more effective and popular. The Authors' previous work, Glaze, also does not seem to be very effective despite dramatic proclamations to the contrary, so I think this might be a case of overhyping an academically interesting but real-world-impractical result. [1]: Courtesy of /u/b3sn0w on Reddit: https://imgur.com/cI7RLAq https://imgur.com/cI7RLAq https://imgur.com/eqe3Dyn https://imgur.com/eqe3Dyn https://imgur.com/1BMASL4 https://imgur.com/1BMASL4

brucethemoose2 3y ago
Yeah. At worst a simple img2img diffusion step would mitigate this, but just eyeballing the examples, traditional denoisers would probably do the job? Denoising is probably a good preprocessing step anyway.
- achileas 3y ago
  It’s a common preprocessing step and I believe that’s how glaze (this lab’s previous work) was defeated.
gedy 3y ago
Maybe it's more about "protecting" images that artists want to publicly share to advertise work, but it's not appropriate for final digital media, etc.
- sesm 3y ago
  In short, anti-AI watermark.
  - johnnyanmac 3y ago
    Yeah. It may mess with the artist's vision but the impact is still way more subtle than other methods used to protect against these unwanted actions. Of course I'm assuming it works to begin with. Sounds like a game of cat and mouse. And AI has a lot of rich cats.
pimlottc 3y ago
I can’t really see any difference in those images on the Twitter example when viewing it on mobile
- pxc 3y ago
  I don't have great vision, but me neither. They're indistinguishable to me (likewise on mobile).
  - Gigachad 3y ago
    I was on desktop and it looks like pretty heavy jpeg compression. Doesn't completely destroy the image, but it's pretty noticeable when blown up large enough.
- milsorgen 3y ago
  It took me a minute too but on the fast you can see some blocky artifacting by the elbow and a few spots elsewhere like curtain upper left.
- Keyframe 3y ago
  look at the green drapes to the right, or any large uniform colored space. It looks similar to bad JPEG artifacts.
- 0xcde4c3db 3y ago
  I didn't see it immediately either, but there's a ton of added noise. The most noticeable bit for me was near the standing person's bent elbow, but there's a lot more that becomes obvious when flipping back and forth between browser tabs instead of swiping on Twitter.
- vhcr 3y ago
  The animation when you change images makes it harder to see the difference, I opened the three images each in its own tab and the differences are more apparent when you change between each other instantly.
  - 3y ago
GaryNumanVevo 3y ago
The artifacts are a non-issue. It's intended images with nightshade are intended to be silently scrapped and avoid human filtering.
- minimaxir 3y ago
  The artifacts are extremely an issue for artists who don't want their images damaged for the possibility of them not being trained by AI. It's a bad tradeoff.
  - GaryNumanVevo 3y ago
    Nightshaded images aren't intended for portfolios. They're mean to be uploaded enmasse and scraped later.
    - AJ007 3y ago
      To where? A place no one sees them and they aren't scraped?
      - filleduchaos 3y ago
        I think the point is that they're akin to a watermark. Even before the current AI boom, plenty of artists have wanted to showcase their work/prove that it exists without necessarily making the highest quality original file public.
      - Diti 3y ago
        Most serious artists I know (at least in my community) release their high-quality images on Patreon or similar.
      - pgeorgi 3y ago
        For example in accounts on image sites that are exposed to suspected scrapers but not to others. Scrapers will still see the real data, but they'll also run into stuff designed to mix up the training process.
kmeisthax 3y ago
The screenshots you sent in [1] are inference, not training. You need to get a Nightshaded image into the training set of an image generator in order for this to have any effect. When you give an image to GPT-4V, Stable Diffusion img2img, or anything else, you're not training the AI - the model is completely frozen and does not change at all[0]. I don't know if anyone else is still scraping new images into the generators. I've heard somewhere that OpenAI stopped scraping around 2021 because they're worried about training on the output of their own models[1]. Adobe Firefly claims to have been trained on Adobe Stock images, but we don't know if Adobe has any particular cutoffs of their own[2]. If you want an image that screws up inference - i.e. one that GPT-4V or Stable Diffusion will choke on - you want an adversarial image. I don't know if you can adversarially train on a model you don't have weights for, though I've heard you can generalize adversarial training against multiple independent models to really screw shit up[3]. [0] All learning capability of text generators come from the fact that they have a context window; but that only provides a short term memory of 2048 tokens. They have no other memory capability. [1] The scenario of what happens when you do this is fancifully called Habsburg AI. The model learns from it's own biases, reinforcing them into stronger biases, while forgetting everything else. [2] It'd be particularly ironic if the only thing Nightshade harms is the one AI generator that tried to be even slightly ethical. [3] At the extremes, these adversarial images fool humans. Though, the study that did this intentionally only showed the images for a small period of time, the idea being that short exposures are akin to a feed-forward neural network with no recurrent computation pathways. If you look at them longer, it's obvious that it's a picture of one thing edited to look like another.
- jerbear4328 3y ago
  [3] sounds really interesting - do you have a link?
  - ittseta 3y ago
    https://www.nature.com/articles/s41467-023-40499-0 https://www.nature.com/articles/s41467-023-40499-0 https://deepmind.google/discover/blog/images-altered-to-trick-machine-vision-can-influence-humans-too/ https://deepmind.google/discover/blog/images-altered-to-tric... Study on the Influence of Adversarial Images on Human Perception
h0p3 3y ago
Sir /u/b3nsn0w is courteous, `/nod`.
kjs3 3y ago
Seems obvious that the people stealing would be adjusting their process to negate these kinds of countermeasures all the time. I don't see this as an arms race the artists are going to win. Not like the LLM folks can consider actually paying their way...the business plan pretty much has "...by stealing everything we can get our hands on..." in the executive summary.

by 542458 3y ago

brucethemoose2 3y ago

Yeah. At worst a simple img2img diffusion step would mitigate this, but just eyeballing the examples, traditional denoisers would probably do the job? Denoising is probably a good preprocessing step anyway.

achileas 3y ago
It’s a common preprocessing step and I believe that’s how glaze (this lab’s previous work) was defeated.

gedy 3y ago

Maybe it's more about "protecting" images that artists want to publicly share to advertise work, but it's not appropriate for final digital media, etc.

sesm 3y ago
In short, anti-AI watermark.
- johnnyanmac 3y ago
  Yeah. It may mess with the artist's vision but the impact is still way more subtle than other methods used to protect against these unwanted actions. Of course I'm assuming it works to begin with. Sounds like a game of cat and mouse. And AI has a lot of rich cats.

pimlottc 3y ago

I can’t really see any difference in those images on the Twitter example when viewing it on mobile

pxc 3y ago
I don't have great vision, but me neither. They're indistinguishable to me (likewise on mobile).
- Gigachad 3y ago
  I was on desktop and it looks like pretty heavy jpeg compression. Doesn't completely destroy the image, but it's pretty noticeable when blown up large enough.
milsorgen 3y ago
It took me a minute too but on the fast you can see some blocky artifacting by the elbow and a few spots elsewhere like curtain upper left.
Keyframe 3y ago
look at the green drapes to the right, or any large uniform colored space. It looks similar to bad JPEG artifacts.
0xcde4c3db 3y ago
I didn't see it immediately either, but there's a ton of added noise. The most noticeable bit for me was near the standing person's bent elbow, but there's a lot more that becomes obvious when flipping back and forth between browser tabs instead of swiping on Twitter.
vhcr 3y ago
The animation when you change images makes it harder to see the difference, I opened the three images each in its own tab and the differences are more apparent when you change between each other instantly.
- 3y ago

GaryNumanVevo 3y ago

The artifacts are a non-issue. It's intended images with nightshade are intended to be silently scrapped and avoid human filtering.

minimaxir 3y ago
The artifacts are extremely an issue for artists who don't want their images damaged for the possibility of them not being trained by AI. It's a bad tradeoff.
- GaryNumanVevo 3y ago
  Nightshaded images aren't intended for portfolios. They're mean to be uploaded enmasse and scraped later.
  - AJ007 3y ago
    To where? A place no one sees them and they aren't scraped?
    - filleduchaos 3y ago
      I think the point is that they're akin to a watermark. Even before the current AI boom, plenty of artists have wanted to showcase their work/prove that it exists without necessarily making the highest quality original file public.
    - Diti 3y ago
      Most serious artists I know (at least in my community) release their high-quality images on Patreon or similar.
    - pgeorgi 3y ago
      For example in accounts on image sites that are exposed to suspected scrapers but not to others. Scrapers will still see the real data, but they'll also run into stuff designed to mix up the training process.

kmeisthax 3y ago

The screenshots you sent in [1] are inference, not training. You need to get a Nightshaded image into the training set of an image generator in order for this to have any effect. When you give an image to GPT-4V, Stable Diffusion img2img, or anything else, you're not training the AI - the model is completely frozen and does not change at all[0]. I don't know if anyone else is still scraping new images into the generators. I've heard somewhere that OpenAI stopped scraping around 2021 because they're worried about training on the output of their own models[1]. Adobe Firefly claims to have been trained on Adobe Stock images, but we don't know if Adobe has any particular cutoffs of their own[2]. If you want an image that screws up inference - i.e. one that GPT-4V or Stable Diffusion will choke on - you want an adversarial image. I don't know if you can adversarially train on a model you don't have weights for, though I've heard you can generalize adversarial training against multiple independent models to really screw shit up[3]. [0] All learning capability of text generators come from the fact that they have a context window; but that only provides a short term memory of 2048 tokens. They have no other memory capability. [1] The scenario of what happens when you do this is fancifully called Habsburg AI. The model learns from it's own biases, reinforcing them into stronger biases, while forgetting everything else. [2] It'd be particularly ironic if the only thing Nightshade harms is the one AI generator that tried to be even slightly ethical. [3] At the extremes, these adversarial images fool humans. Though, the study that did this intentionally only showed the images for a small period of time, the idea being that short exposures are akin to a feed-forward neural network with no recurrent computation pathways. If you look at them longer, it's obvious that it's a picture of one thing edited to look like another.

jerbear4328 3y ago
[3] sounds really interesting - do you have a link?
- ittseta 3y ago
  https://www.nature.com/articles/s41467-023-40499-0 https://www.nature.com/articles/s41467-023-40499-0 https://deepmind.google/discover/blog/images-altered-to-trick-machine-vision-can-influence-humans-too/ https://deepmind.google/discover/blog/images-altered-to-tric... Study on the Influence of Adversarial Images on Human Perception

h0p3 3y ago

Sir /u/b3nsn0w is courteous, `/nod`.

kjs3 3y ago

Seems obvious that the people stealing would be adjusting their process to negate these kinds of countermeasures all the time. I don't see this as an arms race the artists are going to win. Not like the LLM folks can consider actually paying their way...the business plan pretty much has "...by stealing everything we can get our hands on..." in the executive summary.