10 ms·

Could you list the concrete "safety checks" that you think prevents real-world harm? What particular image that you think a random human will ask the AI to gene

by srid 3y ago

Could you list the concrete "safety checks" that you think prevents real-world harm? What particular image that you think a random human will ask the AI to generate, which then leads to concrete harm in the real world?

politician 3y ago
Not even the large companies will explain with precision their implementation of safety. Until then, we must view this “safety” as both a scapegoat and a vector for social engineering.
- astrange 3y ago
  Companies are not going to explain their legal risks in their marketing material.
causal 3y ago
If 1 in 1,000 generations will randomly produce memorized CSAM that slipped into the training set then yeah, it's pretty damn unsafe to use. Producing memorized images has precedent[0]. Is it unlikely? Sure, but worth validating. [0] https://arxiv.org/abs/2301.13188 https://arxiv.org/abs/2301.13188
- srid 3y ago
  Okay, by "safety checks" you meant the already unlawful things like CSAM, but not politically-overloaded beliefs like "diversity"? The latter is what the comment[1] you were replying to was referring to (viz. "considering the recent Gemini debacle"[2]). [1] https://news.ycombinator.com/item?id=39466991 https://news.ycombinator.com/item?id=39466991 [2] https://news.ycombinator.com/item?id=39456577 https://news.ycombinator.com/item?id=39456577
  - causal 3y ago
    Right, by "rather have this [nothing]" I meant Stable Diffusion doing some basic safety checking, not Google's obviously flawed ideas of safety. I should have made that clear. I posed the worst-case scenario of generating actual CSAM in response to your question, "What particular image that you think a random human will ask the AI to generate, which then leads to concrete harm in the real world?"
    - thomquaid 3y ago
      Could you elaborate on the concrete real world harm?
- yreg 3y ago
  Why not run the safety check on the training data?
  - causal 3y ago
    They try to, but it is difficult to comb through billions of images, and at least some of SD's earlier datasets were later found to have been contaminated with CSAM[0]. https://www.404media.co/laion-datasets-removed-stanford-csam-child-abuse/ https://www.404media.co/laion-datasets-removed-stanford-csam...
- dns_snek 3y ago
  Do you have an example? I've never heard of anyone accidentally generating CSAM, with any model. "1 in 1,000" is just an obviously bogus probability, there must have been billions of images generated using hundreds of different models. Besides, and this is a serious question, what's the harm of a model accidentally generating CSAM? If you weren't intending to generate these images then you would just discard the output, no harm done. Nobody is forcing you to use a model that might accidentally offend you with its output. You can try "aligning" it, but you'll just end up with Google Gemini style "Sorry I can't generate pictures of white people".
  - causal 3y ago
    Earlier datasets used by SD were likely contaminated with CSAM[0]. It was unlikely to have been significant enough to result in memorized images, but checking the safety of models increases that confidence. And yeah I think we should care, for a lot of reasons, but a big one is just trying to stay well within the law. [0] https://www.404media.co/laion-datasets-removed-stanford-csam-child-abuse/ https://www.404media.co/laion-datasets-removed-stanford-csam...
    - astrange 3y ago
      SD always removed enough nsfw material that this probably never made it in there.
  - 7moritz7 3y ago
    Then you know almost nothing about the SD 1.5 ecosystem apparently. I've finetuned multiple models myself and it's nearly impossible to get rid of the child-bias in anime-derived models (which applies to 90 % of character focussed models) including nsfw ones. Took me like 30 attempts to get somewhere reasonable and it's still noticeable.
    - dns_snek 3y ago
      If we're being honest, anime and anything "anime-derived" is uncomfortably close to CSAM as a source material, before you even get SD involved, so I'm not surprised. What I had in mind were regular general purpose models which I've played around with quite extensively.
astrange 3y ago
The harm is that any use of the model becomes illegal in most countries (or offends credit card processors) if it easily generates porn. Especially if it does it when you didn't ask for it.
dyslexit 3y ago
This question narrows the scope of "safety" to something less than what the people at SD or even probably what OP cares about. _Non-random_ CSAM requests targeting potentially real people is the obvious answer here, but even non-CSAM sexual content is also a probably a threat. I can understand frustration with it currently going overboard on blurring, but removing safety checks altogether would result in SD mainly being associated with porn pretty quickly, which I'm sure Stability AI wants to avoid for the safety of their company. Add to that, parents who want to avoid having their kids generate sexual content would now need to prevent their kids from using this tool because it can create it randomly, limiting SD usage to kids 18+ (which is probably something else Stability AI does not want to deal with.) It's definitely a balance between going overboard and having restrictions though. I haven't used SD in several months now so I'm not sure where that balance is right now.
- int_19h 3y ago
  > non-CSAM sexual content is also a probably a threat To whom? SD's reputation, perhaps - but that ship has already sailed with 1.x. That aside, why is generated porn threatening? If anything, anti-porn crusaders ought to rejoice, given that it doesn't involve actual humans performing all those acts.
  - dyslexit 3y ago
    As I said, it means parents who don't want their young children seeing porn (whether you agree with them or not) would no longer be able to let their children use SD. I'm not making a statement on what our society should or shouldn't allow, I'm pointing out what _is currently_ the standard in the United States and many other, more socially conservative, countries. SD would become more heavily regulated, an 18+ tool in the US, and potentially banned in other countries. You can have your own opinion on it, but surely you can see the issue here?
    - int_19h 3y ago
      I can definitely see an argument for a "safe" model being available for this scenario. I don't see why all models SD releases should be so neutered, however.

by srid 3y ago

politician 3y ago

Not even the large companies will explain with precision their implementation of safety. Until then, we must view this “safety” as both a scapegoat and a vector for social engineering.

astrange 3y ago
Companies are not going to explain their legal risks in their marketing material.

causal 3y ago

If 1 in 1,000 generations will randomly produce memorized CSAM that slipped into the training set then yeah, it's pretty damn unsafe to use. Producing memorized images has precedent[0]. Is it unlikely? Sure, but worth validating. [0] https://arxiv.org/abs/2301.13188 https://arxiv.org/abs/2301.13188

srid 3y ago
Okay, by "safety checks" you meant the already unlawful things like CSAM, but not politically-overloaded beliefs like "diversity"? The latter is what the comment[1] you were replying to was referring to (viz. "considering the recent Gemini debacle"[2]). [1] https://news.ycombinator.com/item?id=39466991 https://news.ycombinator.com/item?id=39466991 [2] https://news.ycombinator.com/item?id=39456577 https://news.ycombinator.com/item?id=39456577
- causal 3y ago
  Right, by "rather have this [nothing]" I meant Stable Diffusion doing some basic safety checking, not Google's obviously flawed ideas of safety. I should have made that clear. I posed the worst-case scenario of generating actual CSAM in response to your question, "What particular image that you think a random human will ask the AI to generate, which then leads to concrete harm in the real world?"
  - thomquaid 3y ago
    Could you elaborate on the concrete real world harm?
yreg 3y ago
Why not run the safety check on the training data?
- causal 3y ago
  They try to, but it is difficult to comb through billions of images, and at least some of SD's earlier datasets were later found to have been contaminated with CSAM[0]. https://www.404media.co/laion-datasets-removed-stanford-csam-child-abuse/ https://www.404media.co/laion-datasets-removed-stanford-csam...
dns_snek 3y ago
Do you have an example? I've never heard of anyone accidentally generating CSAM, with any model. "1 in 1,000" is just an obviously bogus probability, there must have been billions of images generated using hundreds of different models. Besides, and this is a serious question, what's the harm of a model accidentally generating CSAM? If you weren't intending to generate these images then you would just discard the output, no harm done. Nobody is forcing you to use a model that might accidentally offend you with its output. You can try "aligning" it, but you'll just end up with Google Gemini style "Sorry I can't generate pictures of white people".
- causal 3y ago
  Earlier datasets used by SD were likely contaminated with CSAM[0]. It was unlikely to have been significant enough to result in memorized images, but checking the safety of models increases that confidence. And yeah I think we should care, for a lot of reasons, but a big one is just trying to stay well within the law. [0] https://www.404media.co/laion-datasets-removed-stanford-csam-child-abuse/ https://www.404media.co/laion-datasets-removed-stanford-csam...
  - astrange 3y ago
    SD always removed enough nsfw material that this probably never made it in there.
- 7moritz7 3y ago
  Then you know almost nothing about the SD 1.5 ecosystem apparently. I've finetuned multiple models myself and it's nearly impossible to get rid of the child-bias in anime-derived models (which applies to 90 % of character focussed models) including nsfw ones. Took me like 30 attempts to get somewhere reasonable and it's still noticeable.
  - dns_snek 3y ago
    If we're being honest, anime and anything "anime-derived" is uncomfortably close to CSAM as a source material, before you even get SD involved, so I'm not surprised. What I had in mind were regular general purpose models which I've played around with quite extensively.

astrange 3y ago

The harm is that any use of the model becomes illegal in most countries (or offends credit card processors) if it easily generates porn. Especially if it does it when you didn't ask for it.

dyslexit 3y ago

This question narrows the scope of "safety" to something less than what the people at SD or even probably what OP cares about. _Non-random_ CSAM requests targeting potentially real people is the obvious answer here, but even non-CSAM sexual content is also a probably a threat. I can understand frustration with it currently going overboard on blurring, but removing safety checks altogether would result in SD mainly being associated with porn pretty quickly, which I'm sure Stability AI wants to avoid for the safety of their company. Add to that, parents who want to avoid having their kids generate sexual content would now need to prevent their kids from using this tool because it can create it randomly, limiting SD usage to kids 18+ (which is probably something else Stability AI does not want to deal with.) It's definitely a balance between going overboard and having restrictions though. I haven't used SD in several months now so I'm not sure where that balance is right now.

int_19h 3y ago
> non-CSAM sexual content is also a probably a threat To whom? SD's reputation, perhaps - but that ship has already sailed with 1.x. That aside, why is generated porn threatening? If anything, anti-porn crusaders ought to rejoice, given that it doesn't involve actual humans performing all those acts.
- dyslexit 3y ago
  As I said, it means parents who don't want their young children seeing porn (whether you agree with them or not) would no longer be able to let their children use SD. I'm not making a statement on what our society should or shouldn't allow, I'm pointing out what _is currently_ the standard in the United States and many other, more socially conservative, countries. SD would become more heavily regulated, an 18+ tool in the US, and potentially banned in other countries. You can have your own opinion on it, but surely you can see the issue here?
  - int_19h 3y ago
    I can definitely see an argument for a "safe" model being available for this scenario. I don't see why all models SD releases should be so neutered, however.