5 ms·

You're missing an important vulnerability of this tech. The model was trained on copyrighted material. There are enormous pressures to close and stop this. You

You're missing an important vulnerability of this tech. The model was trained on copyrighted material. There are enormous pressures to close and stop this. You can't predict future legislation. Intellectual property legislation (which is an absolute cancer IMO) can outlaw models and their results. It can outlaw distribution of the data sets, the training, the models. Tech companies already acted way beyond the requirements of the law and effectively censured open source projects like popcorn time. Can they prevent a determined hacker which already got everything? No. Could this be the last model to be trained on a wide dataset, available to the public? Yes. Could they make it a living hell where getting these tools in the future will only be from untrustworthy websites where half the download buttons give you an exe, and all your less tech savvy friends won't bother? Easily. Could this tool become impossible for companies to use without risking litigation? Very easily. People tend to forget those making the rules do not have their interests at heart, and every single intellectual property law is designed to leave companies and not people holding all the rights. And those laws can absolutely do damage. Do not underestimate the power of legislation.

kmeisthax 4y ago
You're thinking copyright liability[0], but the real worry, straight from the mouths of the Stability people[1], is AI-generated CSAM. That will make the whole field of generative art legally radioactive. At least with copyright law, there's an argument for training being fair use. If generative art becomes a notorious market for CSAM, everyone in the field goes to jail. [0] Also, I'd like to know what your opinion is on GitHub Copilot. A lot of people decry Copilot for stealing code but love Stable Diffusion for being public, even though they're the same concept and trained in the same quasi-ethical way. [1] https://www.reddit.com/r/StableDiffusion/comments/y9ga5s/comment/it5kkic/ https://www.reddit.com/r/StableDiffusion/comments/y9ga5s/com...
- diebeforei485 4y ago
  They were shamed into working with a nonprofit aimed at protecting children (Thorn) whose executive director stated publicly at the Stanford conference a few weeks ago that her organization is against the concept of synthetic images.
- pantalaimon 4y ago
  Maybe I’m naive, but isn’t AI generated CSAM a good outcome, actually - because it doesn’t require actual children to be hurt?
  - kortilla 4y ago
    This area is already well explored just with fake CSAM generated by artists using photoshop, cartoons, etc. The modern thinking is that it supports and encourages a behavior that can lead to actual violence. If you constantly watch videos of people eating cheeseburgers, you might want to eat a cheeseburger yourself.
    - diebeforei485 4y ago
      This is like claiming video games cause violence, which is absolutely not the case. More likely people will just generate more synthetic content to consume.
      - thrwyoilarticle 4y ago
        IMO it's more like claiming that video games lead to video game fans and addicts. Which is true.
        diebeforei485 4y ago
        Yeah, people who like looking at synthetic images will have easy access to more synthetic images (and could even generate them on their own machines). But they are not harming anyone else.
        thrwyoilarticle 4y ago
        ...but they're becoming child porn addicts
    - jimbob45 4y ago
      Then why is GTA 5 legal? Or Hannibal? Is it ever possible to trust anyone with self-determination?
- yarg 4y ago
  The child porn problem is a double edged sword. Detection becomes easier - is it pornography with a child in it? Generation starts to become trivial - this video, but this person has the features of an X year old. At least in the latter case no-one's actually getting raped.
- miracle2k 4y ago
  > If generative art becomes a notorious market for CSAM, everyone in the field goes to jail. No one will go to jail, except maybe some people who get caught creating, distributing or collecting those images.
- kenneth 4y ago
  We can likely make the very strong assumption that the training data didn't contain any CSAM so it would be more difficult for the model to produce CSAM. Also, I would imagine they trained the model without porn too, so inferring CSAM based on legal adult porn would also be quite difficult. Am I missing something?
  - zarzavat 4y ago
    These models are intended to converge to the capabilities of human artists and beyond. A human artist is obviously capable of generating CSAM, even if they have never seen that before. Filtering of training data is countered by increasing capabilities to generalize: Two years ago, that was a viable strategy: models could barely produce what was in the training data again. Today models can generalize much better and compose concepts they have been trained on into new concepts that they haven’t. Two years from now, filtering will be irrelevant.
    - d3nj4l 4y ago
      Not only that, but with techniques like in-painting you could start with something that wasn't CP and then progressively make the model generate parts of it which then make up such an image right now. Stability saying they want to release an open version of SD that can't make CP is like a pen maker selling a pen that can't make CP: horseshit.
  - rngname22 4y ago
    Actually I saw Stable Diffusion-generated semi-stylized / semi-photorealistic (kind of like photorealistic-ish anime) CSAM on 4chan literally a day or so ago when I randomly decided to go to 4chan and saw that AI art threads are super popular there right now. Keep in mind, there already are a lot of illustrated/anime style pictures of CSAM on that site of years though (something that is legal in many countries), so it's sort of becoming a blurred area as these AI art generators are still somewhat like that but now are getting to be more photorealistic. As far as the models not being trained on NSFW content, there was already leaked models that were, and there are unofficial models trained by outsiders using SD that are specifically trained on for example adult image websites.
- d3nj4l 4y ago
  What an utterly predictable development. I was happy that Stability put their model out there without any waffling about "concerns" and "communities", but I was always skeptical they'd last. And well, now they're folding like cardboard when faced with a criticism that they should've seen coming. The most concerning thing here is that there's no conceivable approach they can take to prevent CP while keeping their model open; either it is open, and people can use/re-train it to make CP, or it is closed.
- concordDance 4y ago
  Note that generated child porn that depicts no real children is actually legal in much of the world. The UK is more the exception than the rule.
- machina_ex_deus 4y ago
  The way I see things, it all starts from the interests of the participants. Stable diffusion got their publicity from opening their model, but their interests are squeezing a maximal profit from it. And then there is Dall-E and midjourney, with similar incentives. Then there are narratives. They are weaved so that the suggested actions and solutions will somehow fit the interests of the participants. The narrative can be CSAM, it can be copyright of artists and owners of the training set, the narrative can be disinformation. The narrative doesn't care that current laws do not prohibit anything and that it's all legal. The narrative justifies actions the participants wanted to do because of their interests. And finally there are actions. They can push legislations, but that's not the only tool (and yes it's slow). Companies can always comply and cooperate, especially when their interests align. Google itself is a participant, with Imagen. They can create a restrictive policy and kick things off their search engine, because that is in their interests too, not because of a narrative or legislation. Just like they profited in YouTube for every piracy site suppressed. The interests of every single company is stacked against individuals running this at home for free. There are enough narratives to be weaved to justify actions which would stop that. For decades, and in many countries even today, just getting paid to drive someone in your car is illegal, and you need a "taxi license". It doesn't need to make sense. We could end up with required license to use generative AI in 10 years and nobody would bat an eye after decades of propaganda and narratives.
- morpheuskafka 4y ago
  I thought there was a case that "virtual" images were already legal. Does that not apply here because real images are used as the training dataset or something? If no illegal images are used as input, I don't see how the output could be (or should be) illegal. There's no nexus to any real person being harmed.
jrm4 4y ago
I could see this happening, and so it's perhaps important to remember that by this logic, Google should go down as well, to say nothing of Copilot.
tarsinge 4y ago
Is it possible to tell from the weights that a model was trained on copyrighted material?
- lbotos 4y ago
  possibly but you'd need to do a couple of transformations first. https://youtu.be/0_BBRNYInx8?t=85 https://youtu.be/0_BBRNYInx8?t=85 This video (released yesterday) talks about how SD takes an image, converts it to latents. You'd need to decode those latents back to an image representation and scan. (possibly other ways but that's the most straightforward I can come up with although time intensive).
Keyframe 4y ago
All this is extremely interesting to see how it will unfold. You know what's also trained on copyrighted material? Real (human) artists! Not only trained but also constantly draw inspiration directly from copyrighted material.
- lbotos 4y ago
  This is my favorite part of this whole "debate". I think the layperson does not realize how "risky" it is to be an "artist" because of copyright. The entire art tradition is based on copying for study, and then using our brains to convince ourselves that we've "transformed it enough" or "my reference is obscure enough that no one will know." Now we've simplified that process, and more people are exposed to the risk. I hope that the law takes a minute to evaluate the pace of change instead of saying "ITS TOO DANGEROUS, we must BAN IT", but my hopes are low.
  - Keyframe 4y ago
    I couldn't have said it better myself. Pretty much all of art relies on inspiration and "inspiration". Pending few possible cases (I've yet to see) where actual art is lifted and pasted, all I've seen so far (firsthand even) is style lifting and even that not fully. Is style copyrightable?
    - vhold 4y ago
      Some very famous paintings it can almost reproduce, like the Mona Lisa and The Last Supper. It can’t get it quite right but I think it’s close enough to be considered copying. So there might be a copyrighted instance of that, but I haven’t seen one yet.
antihero 4y ago
Surely eventually, hardware will catch up and in ten years people will be able to create models like this on their home PC, and people will just share PBs of images as datatasets/models like with normal piracy?
d110af5ccf 4y ago
> The model was trained on copyrighted material. There are enormous pressures to close and stop this. > getting these tools in the future will only be from untrustworthy websites where half the download buttons give you an exe These models can already be downloaded via well known (ie community reviewed) torrents. So can many terabytes of labeled training data. This particular horse is well out of the barn.
- hda2 4y ago
  My network connection is horrible for large http downloads but torrents work fine. Can you provide guidance on where I can find these torrents? It doesn't have to be direct links, just a hint that can help me find them.
  - d110af5ccf 4y ago
    This NovelAI guide is a good starting point. https://rentry.org/voldy https://rentry.org/voldy If you're interested in training data then reading about what was used and where it was sourced from would be informative. As to torrent indexing services, well, I dunno if I'm supposed to link those here but they're easy enough to find and there are a lot of them out there.

kmeisthax 4y ago

You're thinking copyright liability[0], but the real worry, straight from the mouths of the Stability people[1], is AI-generated CSAM. That will make the whole field of generative art legally radioactive. At least with copyright law, there's an argument for training being fair use. If generative art becomes a notorious market for CSAM, everyone in the field goes to jail. [0] Also, I'd like to know what your opinion is on GitHub Copilot. A lot of people decry Copilot for stealing code but love Stable Diffusion for being public, even though they're the same concept and trained in the same quasi-ethical way. [1] https://www.reddit.com/r/StableDiffusion/comments/y9ga5s/comment/it5kkic/ https://www.reddit.com/r/StableDiffusion/comments/y9ga5s/com...

diebeforei485 4y ago
They were shamed into working with a nonprofit aimed at protecting children (Thorn) whose executive director stated publicly at the Stanford conference a few weeks ago that her organization is against the concept of synthetic images.
pantalaimon 4y ago
Maybe I’m naive, but isn’t AI generated CSAM a good outcome, actually - because it doesn’t require actual children to be hurt?
- kortilla 4y ago
  This area is already well explored just with fake CSAM generated by artists using photoshop, cartoons, etc. The modern thinking is that it supports and encourages a behavior that can lead to actual violence. If you constantly watch videos of people eating cheeseburgers, you might want to eat a cheeseburger yourself.
  - diebeforei485 4y ago
    This is like claiming video games cause violence, which is absolutely not the case. More likely people will just generate more synthetic content to consume.
    - thrwyoilarticle 4y ago
      IMO it's more like claiming that video games lead to video game fans and addicts. Which is true.
      - diebeforei485 4y ago
        Yeah, people who like looking at synthetic images will have easy access to more synthetic images (and could even generate them on their own machines). But they are not harming anyone else.
        thrwyoilarticle 4y ago
        ...but they're becoming child porn addicts
  - jimbob45 4y ago
    Then why is GTA 5 legal? Or Hannibal? Is it ever possible to trust anyone with self-determination?
yarg 4y ago
The child porn problem is a double edged sword. Detection becomes easier - is it pornography with a child in it? Generation starts to become trivial - this video, but this person has the features of an X year old. At least in the latter case no-one's actually getting raped.
miracle2k 4y ago
> If generative art becomes a notorious market for CSAM, everyone in the field goes to jail. No one will go to jail, except maybe some people who get caught creating, distributing or collecting those images.
kenneth 4y ago
We can likely make the very strong assumption that the training data didn't contain any CSAM so it would be more difficult for the model to produce CSAM. Also, I would imagine they trained the model without porn too, so inferring CSAM based on legal adult porn would also be quite difficult. Am I missing something?
- zarzavat 4y ago
  These models are intended to converge to the capabilities of human artists and beyond. A human artist is obviously capable of generating CSAM, even if they have never seen that before. Filtering of training data is countered by increasing capabilities to generalize: Two years ago, that was a viable strategy: models could barely produce what was in the training data again. Today models can generalize much better and compose concepts they have been trained on into new concepts that they haven’t. Two years from now, filtering will be irrelevant.
  - d3nj4l 4y ago
    Not only that, but with techniques like in-painting you could start with something that wasn't CP and then progressively make the model generate parts of it which then make up such an image right now. Stability saying they want to release an open version of SD that can't make CP is like a pen maker selling a pen that can't make CP: horseshit.
- rngname22 4y ago
  Actually I saw Stable Diffusion-generated semi-stylized / semi-photorealistic (kind of like photorealistic-ish anime) CSAM on 4chan literally a day or so ago when I randomly decided to go to 4chan and saw that AI art threads are super popular there right now. Keep in mind, there already are a lot of illustrated/anime style pictures of CSAM on that site of years though (something that is legal in many countries), so it's sort of becoming a blurred area as these AI art generators are still somewhat like that but now are getting to be more photorealistic. As far as the models not being trained on NSFW content, there was already leaked models that were, and there are unofficial models trained by outsiders using SD that are specifically trained on for example adult image websites.
d3nj4l 4y ago
What an utterly predictable development. I was happy that Stability put their model out there without any waffling about "concerns" and "communities", but I was always skeptical they'd last. And well, now they're folding like cardboard when faced with a criticism that they should've seen coming. The most concerning thing here is that there's no conceivable approach they can take to prevent CP while keeping their model open; either it is open, and people can use/re-train it to make CP, or it is closed.
concordDance 4y ago
Note that generated child porn that depicts no real children is actually legal in much of the world. The UK is more the exception than the rule.
machina_ex_deus 4y ago
The way I see things, it all starts from the interests of the participants. Stable diffusion got their publicity from opening their model, but their interests are squeezing a maximal profit from it. And then there is Dall-E and midjourney, with similar incentives. Then there are narratives. They are weaved so that the suggested actions and solutions will somehow fit the interests of the participants. The narrative can be CSAM, it can be copyright of artists and owners of the training set, the narrative can be disinformation. The narrative doesn't care that current laws do not prohibit anything and that it's all legal. The narrative justifies actions the participants wanted to do because of their interests. And finally there are actions. They can push legislations, but that's not the only tool (and yes it's slow). Companies can always comply and cooperate, especially when their interests align. Google itself is a participant, with Imagen. They can create a restrictive policy and kick things off their search engine, because that is in their interests too, not because of a narrative or legislation. Just like they profited in YouTube for every piracy site suppressed. The interests of every single company is stacked against individuals running this at home for free. There are enough narratives to be weaved to justify actions which would stop that. For decades, and in many countries even today, just getting paid to drive someone in your car is illegal, and you need a "taxi license". It doesn't need to make sense. We could end up with required license to use generative AI in 10 years and nobody would bat an eye after decades of propaganda and narratives.
morpheuskafka 4y ago
I thought there was a case that "virtual" images were already legal. Does that not apply here because real images are used as the training dataset or something? If no illegal images are used as input, I don't see how the output could be (or should be) illegal. There's no nexus to any real person being harmed.

jrm4 4y ago

I could see this happening, and so it's perhaps important to remember that by this logic, Google should go down as well, to say nothing of Copilot.

tarsinge 4y ago

Is it possible to tell from the weights that a model was trained on copyrighted material?

lbotos 4y ago
possibly but you'd need to do a couple of transformations first. https://youtu.be/0_BBRNYInx8?t=85 https://youtu.be/0_BBRNYInx8?t=85 This video (released yesterday) talks about how SD takes an image, converts it to latents. You'd need to decode those latents back to an image representation and scan. (possibly other ways but that's the most straightforward I can come up with although time intensive).

Keyframe 4y ago

All this is extremely interesting to see how it will unfold. You know what's also trained on copyrighted material? Real (human) artists! Not only trained but also constantly draw inspiration directly from copyrighted material.

lbotos 4y ago
This is my favorite part of this whole "debate". I think the layperson does not realize how "risky" it is to be an "artist" because of copyright. The entire art tradition is based on copying for study, and then using our brains to convince ourselves that we've "transformed it enough" or "my reference is obscure enough that no one will know." Now we've simplified that process, and more people are exposed to the risk. I hope that the law takes a minute to evaluate the pace of change instead of saying "ITS TOO DANGEROUS, we must BAN IT", but my hopes are low.
- Keyframe 4y ago
  I couldn't have said it better myself. Pretty much all of art relies on inspiration and "inspiration". Pending few possible cases (I've yet to see) where actual art is lifted and pasted, all I've seen so far (firsthand even) is style lifting and even that not fully. Is style copyrightable?
  - vhold 4y ago
    Some very famous paintings it can almost reproduce, like the Mona Lisa and The Last Supper. It can’t get it quite right but I think it’s close enough to be considered copying. So there might be a copyrighted instance of that, but I haven’t seen one yet.

antihero 4y ago

Surely eventually, hardware will catch up and in ten years people will be able to create models like this on their home PC, and people will just share PBs of images as datatasets/models like with normal piracy?

d110af5ccf 4y ago

> The model was trained on copyrighted material. There are enormous pressures to close and stop this. > getting these tools in the future will only be from untrustworthy websites where half the download buttons give you an exe These models can already be downloaded via well known (ie community reviewed) torrents. So can many terabytes of labeled training data. This particular horse is well out of the barn.

hda2 4y ago
My network connection is horrible for large http downloads but torrents work fine. Can you provide guidance on where I can find these torrents? It doesn't have to be direct links, just a hint that can help me find them.
- d110af5ccf 4y ago
  This NovelAI guide is a good starting point. https://rentry.org/voldy https://rentry.org/voldy If you're interested in training data then reading about what was used and where it was sourced from would be informative. As to torrent indexing services, well, I dunno if I'm supposed to link those here but they're easy enough to find and there are a lot of them out there.