4 ms·

The FAQ says: >Once you add a cloak to an image, the same cloak can prevent different AI models (e.g., Midjourney, Stable Diffusion, etc.) from stealing the st

by Imnimo 4y ago

The FAQ says: >Once you add a cloak to an image, the same cloak can prevent different AI models (e.g., Midjourney, Stable Diffusion, etc.) from stealing the style of the cloaked image But my understanding of what they're actually doing is a data poisoning attack on the fine-tuning process (e.g. someone tries to finetune StableDiffusion on a handful of a particular artist's images). Does Midjourney offer that sort of finetuning at all? And isn't it misleading to say that you're preventing Midjourney and Stable Diffusion from "stealing the style"? That seems to imply you're also poisoning the regular training process. I'm also very unconvinced that this offers any meaningful protection in practice. The lesson from years of research on adversarial perturbations has been that it's very easy to make your method look successful in your own paper against a naive adversary, and way harder to make something that stands up to an intelligent counter-attack. I'm not really convinced it's ethical to present this in the way they do to artists who probably don't understand the technical details. Even with their disclaimers and limitations section, this website gives a much rosier picture of the tool's effectiveness than I think it justified. If an artist is concerned enough about this sort thing to want to use this tool, I think they'd be upset to learn about it's true (in)effectiveness, and once they've chosen to post their Glazed images, it's too late.

not-chatgpt 4y ago
The description (as well as the project as a whole) is incredibly disingenuous and capitalized heavily on artist's fear of being trained by AI. Even one pass of SD's img2img with low denoise is enough to bypass this data poisoning attack. It's a useless attack that makes training a tidbit more inconvenient. The project also gained infamy when it stole code from an open source project with GPL license without giving credits. Additionally, SD now offers opt outs for artists and MD likely does not train on these artists at all.
d0mine 4y ago
> against a naive adversary It is all you need in practice otherwise watermarks wouldn't be a thing. You don't need to run faster than a lion, all you need is to run faster than the gazelle next to you.
- Imnimo 4y ago
  Disagree with that. We're talking about defense against a situation where a user downloads a fine-tuning colab notebook, uploads a handful of the artist's pictures, and "steals" their style. The user here is interested in the specific artist, not someone blindly ingesting billions of images. And the technical know-how to overcome the defense will be in the colab notebook, written by an expert, completely hidden from a user.
dragonwriter 4y ago
> But my understanding of what they’re actually doing is a data poisoning attack on the fine-tuning process (e.g. someone tries to finetune StableDiffusion on a handful of a particular artist’s images). No, its poisoning core model training as well, because the core models are trained with scraped art with artist and other metadata, which is why you can use “in the style of…” prompts for lots of artists with the core models. It also applies to fine tuning, etc.
- Imnimo 4y ago
  It is absolutely true that the Glazed images will be scooped up by a fresh webscrape for training data for a new model. But there isn't any evidence that this would provide any actual defense. Their paper only studies the fine-tuning scenario. It seems to me that if you train your text-to-image system from scratch on Glazed images, Glaze has lost its upper-hand. You'd essentially be performing adversarial training but with a fixed adversary! At the very least, I'd want to see some actual experiments on training from scratch before telling artists that Glaze will protect them in that scenario. And I'm very skeptical that it would.
  - dragonwriter 4y ago
    I hadn’t really thought about that. If it doesn’t work against people training the base models, and those are inevitably going to be trained on a wider and wider set of internet-available imagery, it seems like this is even more futile.
- sebzim4500 4y ago
  Wouldn't having a small portion of adverserially modified images in your training set improve the robustness of the model? It's a known technique to do this kind of thing intentionally to train models that are more resistant to adverserial attacks. One reason people don't do this is the cost of running PGD is so high, but in this case your adversaries are doing it for you free of charge.

by Imnimo 4y ago

not-chatgpt 4y ago

The description (as well as the project as a whole) is incredibly disingenuous and capitalized heavily on artist's fear of being trained by AI. Even one pass of SD's img2img with low denoise is enough to bypass this data poisoning attack. It's a useless attack that makes training a tidbit more inconvenient. The project also gained infamy when it stole code from an open source project with GPL license without giving credits. Additionally, SD now offers opt outs for artists and MD likely does not train on these artists at all.

d0mine 4y ago

> against a naive adversary It is all you need in practice otherwise watermarks wouldn't be a thing. You don't need to run faster than a lion, all you need is to run faster than the gazelle next to you.

Imnimo 4y ago
Disagree with that. We're talking about defense against a situation where a user downloads a fine-tuning colab notebook, uploads a handful of the artist's pictures, and "steals" their style. The user here is interested in the specific artist, not someone blindly ingesting billions of images. And the technical know-how to overcome the defense will be in the colab notebook, written by an expert, completely hidden from a user.

dragonwriter 4y ago

> But my understanding of what they’re actually doing is a data poisoning attack on the fine-tuning process (e.g. someone tries to finetune StableDiffusion on a handful of a particular artist’s images). No, its poisoning core model training as well, because the core models are trained with scraped art with artist and other metadata, which is why you can use “in the style of…” prompts for lots of artists with the core models. It also applies to fine tuning, etc.

Imnimo 4y ago
It is absolutely true that the Glazed images will be scooped up by a fresh webscrape for training data for a new model. But there isn't any evidence that this would provide any actual defense. Their paper only studies the fine-tuning scenario. It seems to me that if you train your text-to-image system from scratch on Glazed images, Glaze has lost its upper-hand. You'd essentially be performing adversarial training but with a fixed adversary! At the very least, I'd want to see some actual experiments on training from scratch before telling artists that Glaze will protect them in that scenario. And I'm very skeptical that it would.
- dragonwriter 4y ago
  I hadn’t really thought about that. If it doesn’t work against people training the base models, and those are inevitably going to be trained on a wider and wider set of internet-available imagery, it seems like this is even more futile.
sebzim4500 4y ago
Wouldn't having a small portion of adverserially modified images in your training set improve the robustness of the model? It's a known technique to do this kind of thing intentionally to train models that are more resistant to adverserial attacks. One reason people don't do this is the cost of running PGD is so high, but in this case your adversaries are doing it for you free of charge.