One issue with this is that the fake CSAM wouldn't be the cartoons of a Japanese Manga, it would (or could) be photo realistic. It could be photo realistic of

One issue with this is that the fake CSAM wouldn't be the cartoons of a Japanese Manga, it would (or could) be photo realistic. It could be photo realistic of real children. This is obviously possibly bad because it might fuel or encourage pedophiles, but it also has lots of other negative possibilities too. One example of a bad thing - you could easily imagine an instagram bot that looks for pictures of people with their kids, then uses a Stable Diffusion like model to produce pictures of the people having sex with their kids, or horrible things happening to the kids, and reply to the target account. The bot might threaten to post the pictures and accuse the person of being a pedophile unless the person pays X in bitcoin (or whatever). Or, the bot could just post such pictures for fun. I think we don't know if fake CSAM will have a good or bad effect on pedophiles and, sadly, there is no real way to reliably test that (so far as I know). Fake CSAM might placate pedophiles, or it might whet their appetite. It's hard to know what to do. I think we will eventually get to the point where very good unrestricted image generation models are available to the general public. When that happens there will be chaos - you will live to see man-made horrors beyond your comprehension.