4 ms·

This methodology detected many hundreds of instances of known CSAM in the training set Damn, I knew training data was kind of wild-westy, but I didn't know it

This methodology detected many hundreds of instances of known CSAM in the training set Damn, I knew training data was kind of wild-westy, but I didn't know it was that sloppy.

goles 3y ago
I think to be fair to the study, it’s building off a previous study that demonstrated Diffusion models can produce CG-CSAM. Which seems like it could have some pretty serious consequences, for victims, law enforcement, AI Ethics, and the Justice system. They reached that conclusion then naturally asked, if the model can produce that, how much of it does it need to be trained on to produce it? And they discovered the answer to that is very little. Which could be for any number of reasons that are far beyond how I understand ML.
- elpocko 3y ago
  It doesn't need to be trained on CSAM at all. You can make a 2-pass workflow using Stable Diffusion: start generating normal porn for a few steps, and for the rest generate children. I don't think this can be stopped.
  - deleted 3y ago
    [deleted]
  - causality0 3y ago
    I don't think that will satisfy "the market" for this material. I've spent time as a forum moderator before and have seen perhaps a dozen pieces of genuine CSAM. There's something about the proportions that makes it instantly stomach-turning. The attempts I've seen by AI porn "artists" just look like small women with somewhat large heads and don't have any of that soul-searing effect. If that's what the "customers" are after I think the training set would have to have a significant amount of the real stuff to replicate it accurately.
- akira2501 3y ago
  > Which seems like it could have some pretty serious consequences, for victims, law enforcement, AI Ethics, and the Justice system. What actual difference is there between a machine generating this material or a human being drawing it by hand? It seems like we're living with the "pretty serious consequences" already, and yet, it seems like we've found a way to manage those effectively already. > if the model can produce that, how much of it does it need to be trained on to produce it? And they discovered the answer to that is very little. The answer may very well be "none at all," particularly if these systems can create image fragments by inference and a different system or even a human can assemble them. > are far beyond how I understand ML. If you have a crappy product, get it regulated, it grants it the imprimatur of credibility and it hamstrings your competitors and startups in the space. An understanding of ML may not be required at all to understand this regulatory situation.
  - goles 3y ago
    Obligatory IANAL, just lifting from the study. > What actual difference is there between a machine generating this material or a human being drawing it by hand? Page 8 Section 4 Legal concerns ——— U.S. Code[…]uses a standard prohibiting any visual depiction of CSA that is “virtually indistinguishable” from a minor engaging in sexual conduct. The definition it uses specifically references computer-generated material as being in scope.U.S. Code[…]states that any depiction of a minor that both contains sexually explicit conduct and is obscene can be prosecuted with the same penalties[...] As such, the current status of CG-CSAM appears to be that prosecution is possible for any representation deemed both graphic and obscene,[…]if that material is indistinguishable. Now that CG-CSAM has reached the point that it may[…] test the application of some of these laws may be tested. Realistic CG-CSAM also presents obvious difficulties when it comes to prosecutions for CSAM possession[…] in general, the appearance of a child being abused has been sufficient for prosecution[…]alternatives may need to be tested that do not require positive identification of a real-world victim (or worse, that would require such a victim or their family to testify) > The answer may very well be "none at all," You may be correct but they do cite a number of counter-measures that will need to be tested and studied.
    - 3y ago
makomk 3y ago
While the LAION data set is sloppy in a whole bunch of ways, this is also just a mess in general. The researchers found apparent CSAM hosted on a bunch of mainstream sites - Reddit, Twitter, Blogspot, Wordpress, XHamster and XVideos. Also apparently a whole bunch of "teen models", nudism, and Japanese "junior idol" content which... well, let's just say that a lot more things used to be legal in the relatively recent past that probably shouldn't have been and leave it at that.
duskwuff 3y ago
The article summarizes how LAION was assembled: > This dataset was built by taking a snapshot of the Common Crawl repository, downloading images referenced in the HTML, reading the “alt” attributes of the images and using CLIP interrogation to discard images that did not sufficiently match the captions. In other words, it was based a random subset of the entire public Internet and there was no human in the loop -- and it's not clear how there could ever be, at the scale this project was operating. (LAION-5B contains 5.8B images.) If anything, it's surprising that the Stanford researchers only found a few thousand CSAM images in the data set.

This methodology detected many hundreds of instances of known CSAM in the training set Damn, I knew training data was kind of wild-westy, but I didn't know it was that sloppy.

goles 3y ago

I think to be fair to the study, it’s building off a previous study that demonstrated Diffusion models can produce CG-CSAM. Which seems like it could have some pretty serious consequences, for victims, law enforcement, AI Ethics, and the Justice system. They reached that conclusion then naturally asked, if the model can produce that, how much of it does it need to be trained on to produce it? And they discovered the answer to that is very little. Which could be for any number of reasons that are far beyond how I understand ML.

elpocko 3y ago
It doesn't need to be trained on CSAM at all. You can make a 2-pass workflow using Stable Diffusion: start generating normal porn for a few steps, and for the rest generate children. I don't think this can be stopped.
- deleted 3y ago
  [deleted]
- causality0 3y ago
  I don't think that will satisfy "the market" for this material. I've spent time as a forum moderator before and have seen perhaps a dozen pieces of genuine CSAM. There's something about the proportions that makes it instantly stomach-turning. The attempts I've seen by AI porn "artists" just look like small women with somewhat large heads and don't have any of that soul-searing effect. If that's what the "customers" are after I think the training set would have to have a significant amount of the real stuff to replicate it accurately.
akira2501 3y ago
> Which seems like it could have some pretty serious consequences, for victims, law enforcement, AI Ethics, and the Justice system. What actual difference is there between a machine generating this material or a human being drawing it by hand? It seems like we're living with the "pretty serious consequences" already, and yet, it seems like we've found a way to manage those effectively already. > if the model can produce that, how much of it does it need to be trained on to produce it? And they discovered the answer to that is very little. The answer may very well be "none at all," particularly if these systems can create image fragments by inference and a different system or even a human can assemble them. > are far beyond how I understand ML. If you have a crappy product, get it regulated, it grants it the imprimatur of credibility and it hamstrings your competitors and startups in the space. An understanding of ML may not be required at all to understand this regulatory situation.
- goles 3y ago
  Obligatory IANAL, just lifting from the study. > What actual difference is there between a machine generating this material or a human being drawing it by hand? Page 8 Section 4 Legal concerns ——— U.S. Code[…]uses a standard prohibiting any visual depiction of CSA that is “virtually indistinguishable” from a minor engaging in sexual conduct. The definition it uses specifically references computer-generated material as being in scope.U.S. Code[…]states that any depiction of a minor that both contains sexually explicit conduct and is obscene can be prosecuted with the same penalties[...] As such, the current status of CG-CSAM appears to be that prosecution is possible for any representation deemed both graphic and obscene,[…]if that material is indistinguishable. Now that CG-CSAM has reached the point that it may[…] test the application of some of these laws may be tested. Realistic CG-CSAM also presents obvious difficulties when it comes to prosecutions for CSAM possession[…] in general, the appearance of a child being abused has been sufficient for prosecution[…]alternatives may need to be tested that do not require positive identification of a real-world victim (or worse, that would require such a victim or their family to testify) > The answer may very well be "none at all," You may be correct but they do cite a number of counter-measures that will need to be tested and studied.
  - 3y ago

makomk 3y ago

While the LAION data set is sloppy in a whole bunch of ways, this is also just a mess in general. The researchers found apparent CSAM hosted on a bunch of mainstream sites - Reddit, Twitter, Blogspot, Wordpress, XHamster and XVideos. Also apparently a whole bunch of "teen models", nudism, and Japanese "junior idol" content which... well, let's just say that a lot more things used to be legal in the relatively recent past that probably shouldn't have been and leave it at that.

duskwuff 3y ago

The article summarizes how LAION was assembled: > This dataset was built by taking a snapshot of the Common Crawl repository, downloading images referenced in the HTML, reading the “alt” attributes of the images and using CLIP interrogation to discard images that did not sufficiently match the captions. In other words, it was based a random subset of the entire public Internet and there was no human in the loop -- and it's not clear how there could ever be, at the scale this project was operating. (LAION-5B contains 5.8B images.) If anything, it's surprising that the Stanford researchers only found a few thousand CSAM images in the data set.