5 ms·

Does anyone know if attempts have been made to trick these ML models into reproducing original copyrighted inputs verbatim (edit: or close enough)? Edit: Askin

Does anyone know if attempts have been made to trick these ML models into reproducing original copyrighted inputs verbatim (edit: or close enough)? Edit: Asking about verbatim copies wasn't really a great question. I should have asked about producing things that are "close enough to cause legal trouble" (whether that be due to copyright, trademark, or something else).

Scaevolus 4y ago
That's a known failure mode called "overfitting" or "memorization", where a specific input text is very accurately reproduced. I'm not aware of it occurring for any copyrighted inputs, but it occurs for many famous artworks-- it's nearly impossible to convince Stable Diffusion to style "Mona Lisa" at all.
sdenton4 4y ago
That's not really how memorization in neutral networks works. For classifiers, memorization is more like learning a hash function and a lookup table; no need to store the full image at all. Even for very large models, the weights are a tiny fraction of the size of the original data. It's probably helpful to think of embeddings for generative models in a similar way; it's a very specific embedding function, like a locality sensitive hash, which doesn't require actually storing the data.
- schemescape 4y ago
  Thanks. Yes, I shouldn't have asked about "verbatim" copies -- I should have asked about something more like "close enough to cause legal trouble". Obviously copying verbatim is a violation of copyright, but there must be some threshold of "close enough" that is still problematic. E.g. compressed MP3s of copyrighted songs aren't a verbatim reproduction, but as far as I'm aware they're still covered by copyright. Trademarks are even broader.
  - makapuf 4y ago
    A band playing publicly a reinterpretation of a famous band will get sued even if its not verbatim.
    - schemescape 4y ago
      That’s an even better example.
polygamous_bat 4y ago
One, the diffusion model's possible output space contains every RGB image ever. But two, it cannot ever possibly contain the original inputs verbatim, because (the size of the model)/(the size of the training set) comes out to be something like 0.2 KB per image. Unless it's an incredible compression algorithm, diffusion necessarily have learned something from the input rather than copy-pasting things, as claimed upthread.
- schemescape 4y ago
  I edited my post a while ago, but I shouldn't have ask about "verbatim" copies. See reply to sibling for a more interesting question.
- TeMPOraL 4y ago
  > Unless it's an incredible compression algorithm, diffusion necessarily have learned something from the input rather than copy-pasting things, as claimed upthread. Arguably, "learning" and "compression" are the same thing. In this sense, you can view SD as a compression algorithm where the decoder is the model, the compressed file is the prompt + tweakable params, and there aren't any error checks made, so you can feed random data into the decoder and get something out.
  - int_19h 4y ago
    Or maybe the compressed file is the model, and prompt + tweakable params is a "path" inside that compressed file?
gwern 4y ago
There's been a lot of work on memorization, yes, and you can also do nearest-neighbor lookups in the original data to gutcheck 'memorization'. As usual, the answer is "it's complicated" but for most practical purposes, the answer is 'no': you will get the Mona Lisa if you ask for it, absolutely, but the odds of a randomly generated image being a doppelganger is near-zero. (If you've seen stuff on social media to the contrary, then you may have been misled by various people peddling img2img or 'variation' functions, or prompting for it, or other ways of lying/ignorance.) But you certainly can get things like watermarks without any real memorization. Watermarks have been a nuisance in GANS, for example - the StyleGAN CATS model was filled with watermarks or attempted meme text captions, even though the cats were so nightmarish that they obviously weren't 'plagiarized' so nobody made a big deal about it back then and they understood the GAN had simply learned that watermarks were a thing in many real images and it would try to imitate them where plausible in a sample.
- int_19h 4y ago
  If there's a specific prompt that makes the model produce such an image (without img2img), why shouldn't it count? The question isn't whether the model produces such things randomly, but rather whether it's capable of producing them in principle, even if it requires a very elaborate prompt.
  - gwern 4y ago
    > If there's a specific prompt that makes the model produce such an image (without img2img), why shouldn't it count? There's a specific prompt which makes a human artist produce such an image without img2img too: "Please draw the Mona Lisa". There - you just did it in your head while reading this comment!
    - int_19h 4y ago
      So? Humans aren't copyrightable works. But a bunch of weights that constitutes the model is just data, and that data can very well contain copyrighted works. The question is whether it does in any meaningful sense. And if it can reproduce them verbatim with the right prompt, I don't see how the answer could be "no", anymore so than a password-protected archive of a copyrighted JPEG.

Scaevolus 4y ago

That's a known failure mode called "overfitting" or "memorization", where a specific input text is very accurately reproduced. I'm not aware of it occurring for any copyrighted inputs, but it occurs for many famous artworks-- it's nearly impossible to convince Stable Diffusion to style "Mona Lisa" at all.

sdenton4 4y ago

That's not really how memorization in neutral networks works. For classifiers, memorization is more like learning a hash function and a lookup table; no need to store the full image at all. Even for very large models, the weights are a tiny fraction of the size of the original data. It's probably helpful to think of embeddings for generative models in a similar way; it's a very specific embedding function, like a locality sensitive hash, which doesn't require actually storing the data.

schemescape 4y ago
Thanks. Yes, I shouldn't have asked about "verbatim" copies -- I should have asked about something more like "close enough to cause legal trouble". Obviously copying verbatim is a violation of copyright, but there must be some threshold of "close enough" that is still problematic. E.g. compressed MP3s of copyrighted songs aren't a verbatim reproduction, but as far as I'm aware they're still covered by copyright. Trademarks are even broader.
- makapuf 4y ago
  A band playing publicly a reinterpretation of a famous band will get sued even if its not verbatim.
  - schemescape 4y ago
    That’s an even better example.

polygamous_bat 4y ago

One, the diffusion model's possible output space contains every RGB image ever. But two, it cannot ever possibly contain the original inputs verbatim, because (the size of the model)/(the size of the training set) comes out to be something like 0.2 KB per image. Unless it's an incredible compression algorithm, diffusion necessarily have learned something from the input rather than copy-pasting things, as claimed upthread.

schemescape 4y ago
I edited my post a while ago, but I shouldn't have ask about "verbatim" copies. See reply to sibling for a more interesting question.
TeMPOraL 4y ago
> Unless it's an incredible compression algorithm, diffusion necessarily have learned something from the input rather than copy-pasting things, as claimed upthread. Arguably, "learning" and "compression" are the same thing. In this sense, you can view SD as a compression algorithm where the decoder is the model, the compressed file is the prompt + tweakable params, and there aren't any error checks made, so you can feed random data into the decoder and get something out.
- int_19h 4y ago
  Or maybe the compressed file is the model, and prompt + tweakable params is a "path" inside that compressed file?

gwern 4y ago

There's been a lot of work on memorization, yes, and you can also do nearest-neighbor lookups in the original data to gutcheck 'memorization'. As usual, the answer is "it's complicated" but for most practical purposes, the answer is 'no': you will get the Mona Lisa if you ask for it, absolutely, but the odds of a randomly generated image being a doppelganger is near-zero. (If you've seen stuff on social media to the contrary, then you may have been misled by various people peddling img2img or 'variation' functions, or prompting for it, or other ways of lying/ignorance.) But you certainly can get things like watermarks without any real memorization. Watermarks have been a nuisance in GANS, for example - the StyleGAN CATS model was filled with watermarks or attempted meme text captions, even though the cats were so nightmarish that they obviously weren't 'plagiarized' so nobody made a big deal about it back then and they understood the GAN had simply learned that watermarks were a thing in many real images and it would try to imitate them where plausible in a sample.

int_19h 4y ago
If there's a specific prompt that makes the model produce such an image (without img2img), why shouldn't it count? The question isn't whether the model produces such things randomly, but rather whether it's capable of producing them in principle, even if it requires a very elaborate prompt.
- gwern 4y ago
  > If there's a specific prompt that makes the model produce such an image (without img2img), why shouldn't it count? There's a specific prompt which makes a human artist produce such an image without img2img too: "Please draw the Mona Lisa". There - you just did it in your head while reading this comment!
  - int_19h 4y ago
    So? Humans aren't copyrightable works. But a bunch of weights that constitutes the model is just data, and that data can very well contain copyrighted works. The question is whether it does in any meaningful sense. And if it can reproduce them verbatim with the right prompt, I don't see how the answer could be "no", anymore so than a password-protected archive of a copyrighted JPEG.