6 ms·

And the real questions come out. Seriously though, I imagine this is less a case of whether this specific implementation permits pornography, but whether any p

And the real questions come out. Seriously though, I imagine this is less a case of whether this specific implementation permits pornography, but whether any porn was included in the dataset it was trained on. No matter how good AI is, it only knows what it knows.

metadat 4y ago
Based on the upvotes, inquiring minds _really want to know_!
- indigodaddy 4y ago
  How do you see upvotes? I’ve never had any visibility to upvoted comments (downvoted of course yes), or at least it’s not obvious at all to me and see no indication of it..
  - gpm 4y ago
    You can see the total score (upvotes - downovtes) for your comment where you see the upvote/downvote button for everyone else's comments. Or at least I can.
- Tagbert 4y ago
  I’ve always been curious about that, too. I don’t think I see anyone else’s votes, up or down.
- josephg 4y ago
  It can make porn images, but I haven't been able to get consistently good results. There's probably ways to tweak the parameters to improve things, but I haven't figured that out. It mostly understands what naked people look like, but the images I've generated involve a lot of accidental body horror. You get a lot of people with extra arms, weird eyes, or body parts in the wrong places. The fact that they explicitly removed porn from the training set comes through pretty clearly in the model. I suspect it could be improved a lot with some specialized retraining. As far as I know, nobody has done that work yet.
  - barrkel 4y ago
    In particular it is very confused about genitalia, it tends towards hermaphrodite.
gpm 4y ago
"Stable Diffusion" is referring to one specific set of models, which were trained on a dataset including some porn. The official implementation has a second model that detects pornography, and replaces outputs including it with a picture of this dude https://www.youtube.com/watch?v=dQw4w9WgXcQ https://www.youtube.com/watch?v=dQw4w9WgXcQ (Not kidding). Removing that is a really simple a one line change in the official script.
- nextaccountic 4y ago
  I guess the question is whether this toggle is available in the GUI, or if someone has to edit source code
  - Gigachad 4y ago
    For the CLI version, there was no UI, you have to open the python file and stub out the NSFW check. (trivial though)
- paulgb 4y ago
  > Not kidding I was amused by this when reading the source. Here’s the function that loads the replacement. https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a0f6ee1af8bb9a5d0016ccb27e36dc/scripts/txt2img.py#L79 https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a... It looks like removing line 309 of the same file would disable the check, but I haven’t tried it.
  - artdigital 4y ago
    This is the line to change to disable the nsfw filter: https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a0f6ee1af8bb9a5d0016ccb27e36dc/scripts/txt2img.py#L309 https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...
    - nsfw_hacker 4y ago
      I dont see this line in V2.1
XorNot 4y ago
I mean I don't know why we have to talk around this point: yes, you can definitely generate pornography with SD. Be prepared to see a lot of weird things, because that super-realism really means it just drives right into the uncanny/body horror valley at full speed. But the second point here is also wrong: the whole reason these models are interesting is because they can generate things they haven't seen before - the corpus of knowledge represents some type of abstract understanding of how words relate to things that theoretically does encode a little bit of the mechanisms behind it. For example, it theoretically should be able to reconstruct human like poses it has never seen before provided it has examples of what humans look like and something which transposes to an approximate value - an obvious example in the context of the original question would be building photorealistic versions of a sketched concept (since somewhere in it's model is an axis which traces from "artistic depiction of a human" to "photograph of a human" in terms of style content). Of course, most people aren't very good at drawing realistic human poses - it's a learned skill. But the magic of deep learning is really that eventually it doesn't need to be - we would hopefully be able to train a model which can be easily copied and distributed which represents the skill, and SD is a big step in that direction (whether it's a local maxima remains to be seen - it's dramatic, but is it versatile?)
- boppo1 4y ago
  >it can generate porn It's not good at penises or vaginas, just breasts and butts. I can find what I like pornographically without ai. But the ducking and dodging around nudity and sexuality is childish and tiresome and we ought to discuss this topic as disinterestedly and nonchalantly as we do the other things it excels or struggles at. Penises and vaginas are not somehow more vile than other things. Sure enough, sexual content ought to be age-gated and there is potential for abuse (slapping a person's face onto explicit imagery without their consent isn't cool yall). But are we working toward AGI or not? Because at some point it's gonna have to know about the birds and the bees.
  - eurasiantiger 4y ago
    Are we talking about Artificial God Intelligence?
yieldcrv 4y ago
Its only a question because devs went out of their way to limit it, to everyones surprise Its like if Photoshop broke itself when you tried to modify or create anything nude or provocative, as the default That would just be weird and thats what these AI software devs have done so everyone patches that contrived feature flag, but nobody knows if they patched it
- dsiroker 4y ago
  Fun fact: Photoshop does exactly that if you open images of currency. See: https://helpx.adobe.com/photoshop/cds.html https://helpx.adobe.com/photoshop/cds.html
  - can16358p 4y ago
    As if anyone who really will print money can't open in GIMP.
    - deleted 4y ago
      [deleted]
    - petesergeant 4y ago
      Probably useful to stop 12 year olds from "accidentally" committing felonies though
  - yieldcrv 4y ago
    It used to just give a warning
can16358p 4y ago
It has to be. There was a sub (now closed) called r/UnstableDiffusion, where people shared their porn output from SD. The filter should be easy to remove and there are already people who simply removed the filter.
quitit 4y ago
If you're curious you can see some of the imagery used to train SD here: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... This is just a small fraction of the imagery and it does include pornography. I respect that if you're online and reading HN, you're probably mature enough to handle seeing pornography. So if you're curious to see some of the training data that made it in: choose from the dropdown "-column-" and change it to "punsafe" and set the value in the adjacent field to "1", then press Apply. Obviously this will show pornography on your screen. An article which talks about the imagery and how this browser came about is here: https://waxy.org/2022/08/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator/ https://waxy.org/2022/08/exploring-12-million-of-the-images-...

metadat 4y ago

Based on the upvotes, inquiring minds _really want to know_!

indigodaddy 4y ago
How do you see upvotes? I’ve never had any visibility to upvoted comments (downvoted of course yes), or at least it’s not obvious at all to me and see no indication of it..
- gpm 4y ago
  You can see the total score (upvotes - downovtes) for your comment where you see the upvote/downvote button for everyone else's comments. Or at least I can.
Tagbert 4y ago
I’ve always been curious about that, too. I don’t think I see anyone else’s votes, up or down.
josephg 4y ago
It can make porn images, but I haven't been able to get consistently good results. There's probably ways to tweak the parameters to improve things, but I haven't figured that out. It mostly understands what naked people look like, but the images I've generated involve a lot of accidental body horror. You get a lot of people with extra arms, weird eyes, or body parts in the wrong places. The fact that they explicitly removed porn from the training set comes through pretty clearly in the model. I suspect it could be improved a lot with some specialized retraining. As far as I know, nobody has done that work yet.
- barrkel 4y ago
  In particular it is very confused about genitalia, it tends towards hermaphrodite.

gpm 4y ago

"Stable Diffusion" is referring to one specific set of models, which were trained on a dataset including some porn. The official implementation has a second model that detects pornography, and replaces outputs including it with a picture of this dude https://www.youtube.com/watch?v=dQw4w9WgXcQ https://www.youtube.com/watch?v=dQw4w9WgXcQ (Not kidding). Removing that is a really simple a one line change in the official script.

nextaccountic 4y ago
I guess the question is whether this toggle is available in the GUI, or if someone has to edit source code
- Gigachad 4y ago
  For the CLI version, there was no UI, you have to open the python file and stub out the NSFW check. (trivial though)
paulgb 4y ago
> Not kidding I was amused by this when reading the source. Here’s the function that loads the replacement. https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a0f6ee1af8bb9a5d0016ccb27e36dc/scripts/txt2img.py#L79 https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a... It looks like removing line 309 of the same file would disable the check, but I haven’t tried it.
- artdigital 4y ago
  This is the line to change to disable the nsfw filter: https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a0f6ee1af8bb9a5d0016ccb27e36dc/scripts/txt2img.py#L309 https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...
  - nsfw_hacker 4y ago
    I dont see this line in V2.1

XorNot 4y ago

I mean I don't know why we have to talk around this point: yes, you can definitely generate pornography with SD. Be prepared to see a lot of weird things, because that super-realism really means it just drives right into the uncanny/body horror valley at full speed. But the second point here is also wrong: the whole reason these models are interesting is because they can generate things they haven't seen before - the corpus of knowledge represents some type of abstract understanding of how words relate to things that theoretically does encode a little bit of the mechanisms behind it. For example, it theoretically should be able to reconstruct human like poses it has never seen before provided it has examples of what humans look like and something which transposes to an approximate value - an obvious example in the context of the original question would be building photorealistic versions of a sketched concept (since somewhere in it's model is an axis which traces from "artistic depiction of a human" to "photograph of a human" in terms of style content). Of course, most people aren't very good at drawing realistic human poses - it's a learned skill. But the magic of deep learning is really that eventually it doesn't need to be - we would hopefully be able to train a model which can be easily copied and distributed which represents the skill, and SD is a big step in that direction (whether it's a local maxima remains to be seen - it's dramatic, but is it versatile?)

boppo1 4y ago
>it can generate porn It's not good at penises or vaginas, just breasts and butts. I can find what I like pornographically without ai. But the ducking and dodging around nudity and sexuality is childish and tiresome and we ought to discuss this topic as disinterestedly and nonchalantly as we do the other things it excels or struggles at. Penises and vaginas are not somehow more vile than other things. Sure enough, sexual content ought to be age-gated and there is potential for abuse (slapping a person's face onto explicit imagery without their consent isn't cool yall). But are we working toward AGI or not? Because at some point it's gonna have to know about the birds and the bees.
- eurasiantiger 4y ago
  Are we talking about Artificial God Intelligence?

yieldcrv 4y ago

Its only a question because devs went out of their way to limit it, to everyones surprise Its like if Photoshop broke itself when you tried to modify or create anything nude or provocative, as the default That would just be weird and thats what these AI software devs have done so everyone patches that contrived feature flag, but nobody knows if they patched it

dsiroker 4y ago
Fun fact: Photoshop does exactly that if you open images of currency. See: https://helpx.adobe.com/photoshop/cds.html https://helpx.adobe.com/photoshop/cds.html
- can16358p 4y ago
  As if anyone who really will print money can't open in GIMP.
  - deleted 4y ago
    [deleted]
  - petesergeant 4y ago
    Probably useful to stop 12 year olds from "accidentally" committing felonies though
- yieldcrv 4y ago
  It used to just give a warning

can16358p 4y ago

It has to be. There was a sub (now closed) called r/UnstableDiffusion, where people shared their porn output from SD. The filter should be easy to remove and there are already people who simply removed the filter.

quitit 4y ago

If you're curious you can see some of the imagery used to train SD here: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... This is just a small fraction of the imagery and it does include pornography. I respect that if you're online and reading HN, you're probably mature enough to handle seeing pornography. So if you're curious to see some of the training data that made it in: choose from the dropdown "-column-" and change it to "punsafe" and set the value in the adjacent field to "1", then press Apply. Obviously this will show pornography on your screen. An article which talks about the imagery and how this browser came about is here: https://waxy.org/2022/08/exploring-12-million-of-the-images-used-to-train-stable-diffusions-image-generator/ https://waxy.org/2022/08/exploring-12-million-of-the-images-...