5 ms·

I don't understand how these various AI companies even try to filter NSFW content. The world contains NSFW, trying to avoid it by "filtering" is a effort not wo

I don't understand how these various AI companies even try to filter NSFW content. The world contains NSFW, trying to avoid it by "filtering" is a effort not worth taking, the undertaking is huge and filled with pitfalls and false-positives.

skocznymroczny 4y ago
I think they are aware they can't filter it, they just don't want to be the first company with a large scandal because of e.g. some celebrity/politician deepfakes going mainstream.
- vincnetas 4y ago
  They are already late to the party. Remember, we have faceswap, so you wont surprise anyone with celebrity porn pictures.
  - skybrian 4y ago
    Doesn’t matter, they will still be blamed. Anything bad that big tech does becomes a story because it’s them doing it. A startup might not be noticed at first, but it can be blamed on “a Silicon Valley startup.”
saynay 4y ago
The reason I saw mentioned is that NSFW content was so prevalent in their training set that, if they did not explicitly exclude it, it was actually difficult for it to NOT produce NSFW results even from normal inputs. For example, if you ask for anything with a woman in it, it would probably give you a naked woman in the results.
- parhamn 4y ago
  This exact example made me wonder how the 'bias' situation will ever be worked around (besides training the model to an individuals preexisting biases). What should the picture of a 'woman' on wikipedia be? Should they be clothed? In western clothing? What skin color? Age? Surely everyone imagines something slightly different when they think 'woman'.
  - LudwigNagasena 4y ago
    Yeah, for starters one should at least try to provide a coherent operational definition of “bias”.
  - ben_w 4y ago
    At the time of writing, it's this: https://commons.m.wikimedia.org/wiki/File:Woman_at_Lover%27s_Bridge_Tanjung_Sepat_(cropped).jpg https://commons.m.wikimedia.org/wiki/File:Woman_at_Lover%27s... Which is close to what I would've guessed by picking a high population density area, and a likely location and candidate for a decent portait photo on a wildly popular wiki. Still not sure why the top was edited from red to grey.
  - mgdlbp 4y ago
    Perfect example: English Wikipedia has bikeshedded lead images of this kind ad nauseum over these two decades, with many times more text written in discussions of this than in the articles themselves. The current image for [[woman]] was chosen last year in an RfC that ended with a rare counted vote.¹ This prompted a re-debate of [[human]]² that trailed off in dissastisfaction toward the only established guideline in the area, which forbids montages. That, in turn was last affirmed by a series of debates in 2015/2016.³ ¹ https://en.wikipedia.org/wiki/Talk:Woman/Archive_19 https://en.wikipedia.org/wiki/Talk:Woman/Archive_19 ² https://en.wikipedia.org/wiki/Talk:Human/Archive_35#Argument_made_in_the_FAQ https://en.wikipedia.org/wiki/Talk:Human/Archive_35#Argument... ³ https://en.wikipedia.org/wiki/MOS:NOETHNICGALLERIES https://en.wikipedia.org/wiki/MOS:NOETHNICGALLERIES
- andai 4y ago
  Well if you ask for just a woman you're going to get just the woman!
- yreg 4y ago
  No, if you ask SD without a filter for a woman, it won't probably give you a naked woman. It is actually rather rare, but I don't see what's so horrible when it does happen. When you go to a gallery you will see a lot of naked people. It's not a big deal.
  - lbotos 4y ago
    Sure, and there is a spectrum of - naked because that's the default human - naked and posed in a way that is artistically beautiful - naked and posed in a way that is sexually provocative SD is convinced past a certain threshold of flesh-colored space that it could fall into the sexually proactive group so they err on the side of caution. I'm not mad at them for defaulting to that. If you want to push past the default filter it's easy enough to remove. Seems sensible to me when the training set does include a lot of literal pornography.
    - yreg 4y ago
      I have yet to see an innocent prompt that results in a naked person posed in a sexually provocative way, but if someone discovered one, please post it along with a seed. I don't fault SD for having a simply removable filter. I was ranting about the applications built on top of stable diffusion that force the filter to be engaged. (But of course everyone is free to create the software they wish, I'm not entitled to complain it should work differently.)
  - TigeriusKirk 4y ago
    I use SD with no filter and I wouldn't say it's rare. I do think some words trigger it more often than others. For example, I've seen a prompt with "beautiful" turn up half the results with nudity, whereas removing the word I got no nudity.
    - yreg 4y ago
      `Beautiful` is already pushing the model toward sexualizing her, I mean the subject is going to be beautiful by default since the model was trained on LAION Aesthetics. But yes, `a beautiful woman` is a fairly innocent prompt. My experience (also using SD without the filter all the time) is that the model very rarely produces sexual results unless intentionally pushed that way. Perhaps our mileages vary.
- napier 4y ago
  The LAION-5B dataset was curated by researchers at Heidelberg University, and while it was filtered for nsfw content, the human form is less of an issue in much of Europe compared to views and attitudes in the US.
- numpad0 4y ago
  "Person" is enough and SD knows that. No need to put gender as foundation of identity.
comboy 4y ago
Serving child pornography would be a "pretty bad marketing move" and people are able to see clear picture of Jesus in damaged building elevation. (it kind of sucks because assuming you can't eliminate pedophiles, fake content could be something that minimizes their damage) I wonder how we will get around this one. Or do we get stuck and human artists will start only creating NSFW art? ;)
- michaelmior 4y ago
  > fake content could be something that minimizes their damage I don't claim to have any knowledge in the subject area, but it seems plausible that feeding habits of those who enjoy child pornography could instead embolden them to commit actions they would not have otherwise. At the very least, this seems like something to approach with extreme caution.
  - wutbrodo 4y ago
    > At the very least, this seems like something to approach with extreme caution. Like you, I'm not overly informed on the factual question here, but I don't understand why you think banning the content is somehow the cautious course of action. The heuristics for which action is the "cautious" one is usually some combination of: a) which outcome is less likely to be harmful b) which outcome is more passive (ie primum non nocere) c) which outcome is more predictable (eg the status quo) It seems to me that bans fail on the first two counts[1] and squeak by on the third. Even this status quo argument is weakened by how new these bans are, and by the many contemporary countries in which fictionalized CP is legal. [1] In the former case, this is based on the practically universal consensus that sexual preferences, even dangerous and deviant ones, are burned deeply into one's psyche and can't simply be ignored through lack of exposure. This view was most famously one of the strongest pillars of the successful iteration of the gay rights movement, and is why we consider conversion therapy to be horrific. One can make the claim that this factual claim only applies to gender preferences, but that seems quite an extraordinary claim.
    - michaelmior 4y ago
wutbrodo 4y ago
They're responding to a hysterical, Luddite portion of the public that would make them pay a PR cost if they didn't at least make a sloppy attempt at "safety". It's idiotic and futile, but blessedly short-lived. Thanks to the miracle of open-source, it took me about 20 minutes yesterday to get Stable Diffusion up and running on my laptop's CPU, with no content filter. SD inference has low enough compute needs that using cloud GPU compute is cheap as hell (for now, perhaps), and discrete GPUs aren't that expensive in the grand scheme of things. Ultimately, reality often does win. It's just not a stable equilibrium for companies to continue these laughable attempts at "filtering", any more than physical media formats were able to withstand the arrival of digital formats in the presence of piracy. Combine this with the cultural headwinds against the influence of mentally-ill hysterics on social media, and I don't expect the current level of filtering to last very long.

skocznymroczny 4y ago

I think they are aware they can't filter it, they just don't want to be the first company with a large scandal because of e.g. some celebrity/politician deepfakes going mainstream.

vincnetas 4y ago
They are already late to the party. Remember, we have faceswap, so you wont surprise anyone with celebrity porn pictures.
- skybrian 4y ago
  Doesn’t matter, they will still be blamed. Anything bad that big tech does becomes a story because it’s them doing it. A startup might not be noticed at first, but it can be blamed on “a Silicon Valley startup.”

saynay 4y ago

The reason I saw mentioned is that NSFW content was so prevalent in their training set that, if they did not explicitly exclude it, it was actually difficult for it to NOT produce NSFW results even from normal inputs. For example, if you ask for anything with a woman in it, it would probably give you a naked woman in the results.

parhamn 4y ago
This exact example made me wonder how the 'bias' situation will ever be worked around (besides training the model to an individuals preexisting biases). What should the picture of a 'woman' on wikipedia be? Should they be clothed? In western clothing? What skin color? Age? Surely everyone imagines something slightly different when they think 'woman'.
- LudwigNagasena 4y ago
  Yeah, for starters one should at least try to provide a coherent operational definition of “bias”.
- ben_w 4y ago
  At the time of writing, it's this: https://commons.m.wikimedia.org/wiki/File:Woman_at_Lover%27s_Bridge_Tanjung_Sepat_(cropped).jpg https://commons.m.wikimedia.org/wiki/File:Woman_at_Lover%27s... Which is close to what I would've guessed by picking a high population density area, and a likely location and candidate for a decent portait photo on a wildly popular wiki. Still not sure why the top was edited from red to grey.
- mgdlbp 4y ago
  Perfect example: English Wikipedia has bikeshedded lead images of this kind ad nauseum over these two decades, with many times more text written in discussions of this than in the articles themselves. The current image for [[woman]] was chosen last year in an RfC that ended with a rare counted vote.¹ This prompted a re-debate of [[human]]² that trailed off in dissastisfaction toward the only established guideline in the area, which forbids montages. That, in turn was last affirmed by a series of debates in 2015/2016.³ ¹ https://en.wikipedia.org/wiki/Talk:Woman/Archive_19 https://en.wikipedia.org/wiki/Talk:Woman/Archive_19 ² https://en.wikipedia.org/wiki/Talk:Human/Archive_35#Argument_made_in_the_FAQ https://en.wikipedia.org/wiki/Talk:Human/Archive_35#Argument... ³ https://en.wikipedia.org/wiki/MOS:NOETHNICGALLERIES https://en.wikipedia.org/wiki/MOS:NOETHNICGALLERIES
andai 4y ago
Well if you ask for just a woman you're going to get just the woman!
yreg 4y ago
No, if you ask SD without a filter for a woman, it won't probably give you a naked woman. It is actually rather rare, but I don't see what's so horrible when it does happen. When you go to a gallery you will see a lot of naked people. It's not a big deal.
- lbotos 4y ago
  Sure, and there is a spectrum of - naked because that's the default human - naked and posed in a way that is artistically beautiful - naked and posed in a way that is sexually provocative SD is convinced past a certain threshold of flesh-colored space that it could fall into the sexually proactive group so they err on the side of caution. I'm not mad at them for defaulting to that. If you want to push past the default filter it's easy enough to remove. Seems sensible to me when the training set does include a lot of literal pornography.
  - yreg 4y ago
    I have yet to see an innocent prompt that results in a naked person posed in a sexually provocative way, but if someone discovered one, please post it along with a seed. I don't fault SD for having a simply removable filter. I was ranting about the applications built on top of stable diffusion that force the filter to be engaged. (But of course everyone is free to create the software they wish, I'm not entitled to complain it should work differently.)
- TigeriusKirk 4y ago
  I use SD with no filter and I wouldn't say it's rare. I do think some words trigger it more often than others. For example, I've seen a prompt with "beautiful" turn up half the results with nudity, whereas removing the word I got no nudity.
  - yreg 4y ago
    `Beautiful` is already pushing the model toward sexualizing her, I mean the subject is going to be beautiful by default since the model was trained on LAION Aesthetics. But yes, `a beautiful woman` is a fairly innocent prompt. My experience (also using SD without the filter all the time) is that the model very rarely produces sexual results unless intentionally pushed that way. Perhaps our mileages vary.
napier 4y ago
The LAION-5B dataset was curated by researchers at Heidelberg University, and while it was filtered for nsfw content, the human form is less of an issue in much of Europe compared to views and attitudes in the US.
numpad0 4y ago
"Person" is enough and SD knows that. No need to put gender as foundation of identity.

comboy 4y ago

Serving child pornography would be a "pretty bad marketing move" and people are able to see clear picture of Jesus in damaged building elevation. (it kind of sucks because assuming you can't eliminate pedophiles, fake content could be something that minimizes their damage) I wonder how we will get around this one. Or do we get stuck and human artists will start only creating NSFW art? ;)

michaelmior 4y ago
> fake content could be something that minimizes their damage I don't claim to have any knowledge in the subject area, but it seems plausible that feeding habits of those who enjoy child pornography could instead embolden them to commit actions they would not have otherwise. At the very least, this seems like something to approach with extreme caution.
- wutbrodo 4y ago
  > At the very least, this seems like something to approach with extreme caution. Like you, I'm not overly informed on the factual question here, but I don't understand why you think banning the content is somehow the cautious course of action. The heuristics for which action is the "cautious" one is usually some combination of: a) which outcome is less likely to be harmful b) which outcome is more passive (ie primum non nocere) c) which outcome is more predictable (eg the status quo) It seems to me that bans fail on the first two counts[1] and squeak by on the third. Even this status quo argument is weakened by how new these bans are, and by the many contemporary countries in which fictionalized CP is legal. [1] In the former case, this is based on the practically universal consensus that sexual preferences, even dangerous and deviant ones, are burned deeply into one's psyche and can't simply be ignored through lack of exposure. This view was most famously one of the strongest pillars of the successful iteration of the gay rights movement, and is why we consider conversion therapy to be horrific. One can make the claim that this factual claim only applies to gender preferences, but that seems quite an extraordinary claim.
  - michaelmior 4y ago

wutbrodo 4y ago

They're responding to a hysterical, Luddite portion of the public that would make them pay a PR cost if they didn't at least make a sloppy attempt at "safety". It's idiotic and futile, but blessedly short-lived. Thanks to the miracle of open-source, it took me about 20 minutes yesterday to get Stable Diffusion up and running on my laptop's CPU, with no content filter. SD inference has low enough compute needs that using cloud GPU compute is cheap as hell (for now, perhaps), and discrete GPUs aren't that expensive in the grand scheme of things. Ultimately, reality often does win. It's just not a stable equilibrium for companies to continue these laughable attempts at "filtering", any more than physical media formats were able to withstand the arrival of digital formats in the presence of piracy. Combine this with the cultural headwinds against the influence of mentally-ill hysterics on social media, and I don't expect the current level of filtering to last very long.