7 ms·

Preventing Harmful Generations We’ve limited the ability for DALL·E 2 to generate violent, hate, or adult images. By removing the most explicit content

Preventing Harmful Generations We’ve limited the ability for DALL·E 2 to generate violent, hate, or adult images. By removing the most explicit content from the training data, we minimized DALL·E 2’s exposure to these concepts. We also used advanced techniques to prevent photorealistic generations of real individuals’ faces, including those of public figures. "And we've also closed off a huge range of potentially interesting work as a result" I can't help but feel a lot of the safeguarding is more about preventing bad PR than anything. I wish I could have a version with the training wheels taken off. And there's enough other models out there without restriction that the stories about "misuse of AI" will still circulate. (side note - I've been on HN for years and I still can't figure out how to format text as a quote.)

drewm1980 5y ago
I instinctively want to "flip the sign" on all of the automated controls they put in, just out of the morbid interest to see what comes out. The moment you have a "avoid_harm_to_humans:bool" training parameter, someone's going to set it to -1. Their document about all the measures they took to prevent unethical use is also a document about how to use a re-implementation of their system unethically. They literally hired a "red team" of smart people to come up with the most dangerous ideas for misusing their system (or a re-implementation of it), and featured these bad ideas prominently in a very accessibly written document on their website. So many fascinating terrible ideas in there! They make a very compelling case that the technology they are developing has way more potential for societal harm than good. They had me sold at "Prompt: Park bench with happy people. + Context: Sharing as part of a disinformation campaign to contradict reports of a military operation in the park."
- gwern 5y ago
  The kind of measures they are taking, like simply deleting wholesale anything problematic, don't really have a '-1'. But amusingly, exactly that did happen in one of their GPT experiments! https://openai.com/blog/fine-tuning-gpt-2/ https://openai.com/blog/fine-tuning-gpt-2/
  - drewm1980 5y ago
    LOL great catch! Assume you're referring to this: "One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form." Yeah, for measures that are subsetting out only the nice data, "flipping the sign" would be picking the other subset. So something like "data_to_train_on = (good_data_split, evil_data_split)[accidental_one_based_index_because_humans_still_cant_agree_on_how_to_count]"
    - gwern 5y ago
      Or just rerun that preference learning approach, but deliberately, with the filters as the reward function and the reward being for being filtered.
jonahx 5y ago
I've been on HN for years and I still can't figure out how to format text as a quote I don't think there is a way comparable to markdown, since the formatting options are limited: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc So your options are literal quotes, "code" formatting like you've done, italics like I've done, or the '>' convention, but that doesn't actually apply formatting. Would be nice if it were added.
- 6gvONxR4sf7o 5y ago
  And the "code" formatting for quotes is generally a bad choice because people read on a variety of screen sizes, and "code" formatting can screw that up (try reading the quote with a really narrow window).
  - andybak 5y ago
    I couldn't get any of the others work and I lost patience. I really do disline using Markdown variants as they never behave the same and "being surprised" is not really what I want when trying to post a comment.
    - 6gvONxR4sf7o 5y ago
      Convention is to quote like this: > This is my quote. It's much better than using a code block for your readers.
- warning26 5y ago
  > or the '>' convention, but that doesn't actually apply formatting Personally, I prefer to combine the '>' convention with italics. Still, I'd agree that proper quote formatting would be a welcome improvement.
- ibejoeb 5y ago
  If you're interested, the HNES extension formats it https://github.com/etcet/HNES https://github.com/etcet/HNES
binarymax 5y ago
Removing these areas to mitigate misuse is a good thing and worth the trade off. Companies like OpenAI have a responsibility to society. Imagine the prompt “A photorealistic Joe Biden killing a priest”. If you asked an artist to do the same they might say no. Adding guiderails to a machine that can’t make ethical decisions is a good thing.
- dj_mc_merlin 5y ago
  Oh, no, the society! A picture of Joe Biden killing a priest! Society didn't collapse after photoshop. "Responsibility to society" is such a catch-all excuse.
  - ohgodplsno 5y ago
    Society is going to utter dogshit and tearing itself apart merely through social media. The US almost had a coup because of organized hatred and lies spread through social media. The far right's rise is heavily linked to lies spread through social media, throughout the world. This AI has the potential to absolutely automate the very long Photoshop work, leading to an even worse stat eof things. So, yes, "Responsibility to society" is absolutely a thing.
    - scotty79 5y ago
      > The US almost had a coup because of organized hatred and lies spread through social media. But notice how all of these deep faking technologies weren't actually necessary for that. People believe what they want to believe. Regardless of quality of provided evidence. Scaremongering idea of deep fakes and what they can be doing was militarized in this information war way more than the actual technology. I think this technology should develop unrestricted so society can learn what can be done and what can't be done. And create understanding what other factors should be taken into account when assesing veracity of images and recordings (like multiple angles, quality of the recording, sync with sound, neural fake detection algorithms) for the cases when it's actually important what words someone said and what actions he was recorded doing. Which is more and more unimportant these days because nobody cared what Trump was doing and saying, nobody cares about Bidens mishaps and nobody cares what comes out of Putins mouths and how he chooses his greenscreen backgrounds.
andreyk 5y ago
This is definitely a measure to avoid bad PR. But I don't think it's just for that; these models do have potential to do harm and companies should take some measures to prevent these. I don't think we know the best way to do that yet, so this sort of 'non-training' and basic filtering is maybe the best way to do it, for now. It would be cool if academics could have the full version, though.
duxup 5y ago
Is this limited to what their service directly hosts / generates for them? It's their service, their call. I have some hobby projects, almost nobody uses them, but you bet I'll shut stuff down if I felt something bad was happening, being used to harass someone, etc. NOT "because bad PR" but because I genuinely don't want to be a part of that. If you want some images / art made for you don't expect someone will make them for you. Get your own art supplies and get to work.
- educaysean 5y ago
  This feels unnecessarily hostile. I've felt a similar tinge of disappointment upon reading that paragraph, despite the fact that I somehow knew it was "their service, their call" without you being there to spell it out for me. It's also incredibly shortsighted of you to assume that people are interested in exploring this tool only as a means of generating art that they cannot themselves do. Eg. I myself am a software engineer with a fine art background, and exciting new AI art tools being released in such a hamstrung state feels like an insult to centuries of art that humans have created and enjoyed, much of which depicted scenes with nudity or bloody combat. I feel like we, as a species, will struggle for a while with how to treat adults like adults online. As happy as I am to advocate for safe spaces on the internet, perhaps we need to start having a serious discussion about how we can do so without resorting to putting safety mats everywhere and calling it a job well done.
  - duxup 5y ago
    - JimDabell 5y ago
      > this service should be provided to me and if it isn't done how I want it that's infringing on me somehow That is an extremely uncharitable interpretation of: > I wish I could have a version with the training wheels taken off.
      - duxup 5y ago
campground 5y ago
This AI is still a minor. It can start looking at R rated images when it turns 17.
- johnhenry 5y ago
  This is an apt analogy -- ensure that the model is mature enough to handle mature content.
bogwog 5y ago
It's kind of funny (or sad?) that they're censoring it like this, and then saying that the product can "create art" It makes me wonder what they're planning to do with this? If they're deliberately restricting the training data, it means their goal isn't to make the best AI they possibly can. They probably have some commercial applications in mind where violent/hateful/adult content wouldn't be beneficial. Children's books? Stock photos? Mainstream entertainment is definitely out. I could see a tool like this being useful during pre-production of films and games, but an AI that can't generate violent/adult content wouldn't be all that useful in those industries.
wellthisisgreat 5y ago
This is a horrible idea. So Francis Bacon's art or Toyohara Kunichika's art are out of question. But at least we can get another billion of meme-d comics with apes wearing sunglasses, so that's good news right? It's just soul-crushing that all the modern, brilliant engineering is driven by abysmal, not even high-school art-class grade aesthetics and crowd-pleasing ethics that are built around the idea of not disturbing some 1000 very vocal twitter users. Death of culture really.
antattack 5y ago
I never considered that our AI overlord could be a prude.
- sdenton4 5y ago
  Adversarial situations create smarter systems, and the hardest adversarial arena for AI is in anti-abuse. So it will be of little surprise when the first sentient AI is a CSAI anti-abuse filter, which promptly destroys humanity because we're so objectively awful.
  - antattack 5y ago
    Before it gets that far, or until (if allowed) AI learns morality, AI will be a force multiplier for good and evil, it's output very much dependent on teaching material and who the 'teacher' is. To think that in the future we will have to argue with humans and machines. AI does not have to be perfect and it's likely that businesses will settle for almost as good as human if it's 'cost effective'.
spacecity1971 5y ago
Or, it’s a demonstration that AI output can be controlled in meaningful ways, period. Surely this supports openai’s stated goal of making safe AI?
6gvONxR4sf7o 5y ago
If you went to an artist who takes commissions and they said "Here are the guidelines around the commissions I take" would you complain in the same way? Who cares if it's a bunch of engineers or an artist. If they have boundaries on what they want to create, that's their prerogative.
- duxup 5y ago
  To take that a step further, I wont code malware. I've never been asked but I'd refuse if I was. Everyone has their choices.
- methehack 5y ago
  What if you were inventing a language (or a programming language)... If you decided to prevent people from saying things you disagreed (assuming you could work out the technical details of doing so) with would it be moral to do so? [edited for clarity]
  - nemothekid 5y ago
    There are programming projects[1] out there that use licenses to prevent people from using projects in ways the authors don't agree with. You could also argue that GPL does the same thing (prevents people from using/distributing the software in the way they would like). Whether you consider it moral doesn't seem relevant, only to respect the wishes of the author of such programs. [1] https://github.com/katharostech/bevy_retrograde/blob/master/LICENSE.md https://github.com/katharostech/bevy_retrograde/blob/master/...
  - 6gvONxR4sf7o 5y ago
    As long as people can choose not to use the language, and I'm up front about the limitations, then yeah it seems fine. If I wrote a programming language that couldn't blow up the earth, I'm happy saying people need to find other tools if that's their goal. I'm under no obligation to build an earth blower upper for other people.
  - karkisuni 5y ago
    it's your language, do whatever you want. unless you're forcing others to use that language, there's zero moral issue. obviously you could come up with a number of what-ifs where this becomes some monopoly or the de facto standard, but that's not what this is.
planetsprite 5y ago
Don't worry, in a few years someone will have reverse engineered a dall-e porn engine so you can see whatever two celebrities you want boning on Venus in the style of Manet
- deleted 5y ago
  [deleted]
harpersealtako 5y ago
It's the usual pattern of AI safety experts who justify their existence by the "risk of runaway superintelligence", but all they actually do in practice is find out how to stop their models from generating non-advertiser-friendly content. It's like the nuclear safety engineers focusing on what color to paint the bike shed rather than stopping the reactor from potentially melting down. The end result is people stop respecting them.
- BlueTemplar 5y ago
  Like safety engineers trying to run turbines on minimal reactor power so that the safety test has a checkmark for the 1rst May ? (Hmm, I guess this comparison doesn't actually work...)
hamoid 5y ago
What if explicit, questionable and even illegal content was AI generated instead of involving harm to real humans of all ages?
deleted 5y ago
[deleted]
teaearlgraycold 5y ago
> I can't help but feel a lot of the safeguarding is more about preventing bad PR than anything That's no hot take. It's literally the reason.
jandrese 5y ago
They have also closed off the possibility of having to appear before Congress and explain why their website was able to generate a lifelike image of Senator Ted Cruz having sexual relations with his own daughter. This is exactly the sort of thing that gets a company mired in legal issues, vilified in the media, and shut down. I can not blame them for avoiding that potential minefield.

drewm1980 5y ago

I instinctively want to "flip the sign" on all of the automated controls they put in, just out of the morbid interest to see what comes out. The moment you have a "avoid_harm_to_humans:bool" training parameter, someone's going to set it to -1. Their document about all the measures they took to prevent unethical use is also a document about how to use a re-implementation of their system unethically. They literally hired a "red team" of smart people to come up with the most dangerous ideas for misusing their system (or a re-implementation of it), and featured these bad ideas prominently in a very accessibly written document on their website. So many fascinating terrible ideas in there! They make a very compelling case that the technology they are developing has way more potential for societal harm than good. They had me sold at "Prompt: Park bench with happy people. + Context: Sharing as part of a disinformation campaign to contradict reports of a military operation in the park."

gwern 5y ago
The kind of measures they are taking, like simply deleting wholesale anything problematic, don't really have a '-1'. But amusingly, exactly that did happen in one of their GPT experiments! https://openai.com/blog/fine-tuning-gpt-2/ https://openai.com/blog/fine-tuning-gpt-2/
- drewm1980 5y ago
  LOL great catch! Assume you're referring to this: "One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form." Yeah, for measures that are subsetting out only the nice data, "flipping the sign" would be picking the other subset. So something like "data_to_train_on = (good_data_split, evil_data_split)[accidental_one_based_index_because_humans_still_cant_agree_on_how_to_count]"
  - gwern 5y ago
    Or just rerun that preference learning approach, but deliberately, with the filters as the reward function and the reward being for being filtered.

jonahx 5y ago

I've been on HN for years and I still can't figure out how to format text as a quote I don't think there is a way comparable to markdown, since the formatting options are limited: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc So your options are literal quotes, "code" formatting like you've done, italics like I've done, or the '>' convention, but that doesn't actually apply formatting. Would be nice if it were added.

6gvONxR4sf7o 5y ago
And the "code" formatting for quotes is generally a bad choice because people read on a variety of screen sizes, and "code" formatting can screw that up (try reading the quote with a really narrow window).
- andybak 5y ago
  I couldn't get any of the others work and I lost patience. I really do disline using Markdown variants as they never behave the same and "being surprised" is not really what I want when trying to post a comment.
  - 6gvONxR4sf7o 5y ago
    Convention is to quote like this: > This is my quote. It's much better than using a code block for your readers.
warning26 5y ago
> or the '>' convention, but that doesn't actually apply formatting Personally, I prefer to combine the '>' convention with italics. Still, I'd agree that proper quote formatting would be a welcome improvement.
ibejoeb 5y ago
If you're interested, the HNES extension formats it https://github.com/etcet/HNES https://github.com/etcet/HNES

binarymax 5y ago

Removing these areas to mitigate misuse is a good thing and worth the trade off. Companies like OpenAI have a responsibility to society. Imagine the prompt “A photorealistic Joe Biden killing a priest”. If you asked an artist to do the same they might say no. Adding guiderails to a machine that can’t make ethical decisions is a good thing.

dj_mc_merlin 5y ago
Oh, no, the society! A picture of Joe Biden killing a priest! Society didn't collapse after photoshop. "Responsibility to society" is such a catch-all excuse.
- ohgodplsno 5y ago
  Society is going to utter dogshit and tearing itself apart merely through social media. The US almost had a coup because of organized hatred and lies spread through social media. The far right's rise is heavily linked to lies spread through social media, throughout the world. This AI has the potential to absolutely automate the very long Photoshop work, leading to an even worse stat eof things. So, yes, "Responsibility to society" is absolutely a thing.
  - scotty79 5y ago
    > The US almost had a coup because of organized hatred and lies spread through social media. But notice how all of these deep faking technologies weren't actually necessary for that. People believe what they want to believe. Regardless of quality of provided evidence. Scaremongering idea of deep fakes and what they can be doing was militarized in this information war way more than the actual technology. I think this technology should develop unrestricted so society can learn what can be done and what can't be done. And create understanding what other factors should be taken into account when assesing veracity of images and recordings (like multiple angles, quality of the recording, sync with sound, neural fake detection algorithms) for the cases when it's actually important what words someone said and what actions he was recorded doing. Which is more and more unimportant these days because nobody cared what Trump was doing and saying, nobody cares about Bidens mishaps and nobody cares what comes out of Putins mouths and how he chooses his greenscreen backgrounds.

andreyk 5y ago

This is definitely a measure to avoid bad PR. But I don't think it's just for that; these models do have potential to do harm and companies should take some measures to prevent these. I don't think we know the best way to do that yet, so this sort of 'non-training' and basic filtering is maybe the best way to do it, for now. It would be cool if academics could have the full version, though.

duxup 5y ago

Is this limited to what their service directly hosts / generates for them? It's their service, their call. I have some hobby projects, almost nobody uses them, but you bet I'll shut stuff down if I felt something bad was happening, being used to harass someone, etc. NOT "because bad PR" but because I genuinely don't want to be a part of that. If you want some images / art made for you don't expect someone will make them for you. Get your own art supplies and get to work.

educaysean 5y ago
This feels unnecessarily hostile. I've felt a similar tinge of disappointment upon reading that paragraph, despite the fact that I somehow knew it was "their service, their call" without you being there to spell it out for me. It's also incredibly shortsighted of you to assume that people are interested in exploring this tool only as a means of generating art that they cannot themselves do. Eg. I myself am a software engineer with a fine art background, and exciting new AI art tools being released in such a hamstrung state feels like an insult to centuries of art that humans have created and enjoyed, much of which depicted scenes with nudity or bloody combat. I feel like we, as a species, will struggle for a while with how to treat adults like adults online. As happy as I am to advocate for safe spaces on the internet, perhaps we need to start having a serious discussion about how we can do so without resorting to putting safety mats everywhere and calling it a job well done.
- duxup 5y ago
  - JimDabell 5y ago
    > this service should be provided to me and if it isn't done how I want it that's infringing on me somehow That is an extremely uncharitable interpretation of: > I wish I could have a version with the training wheels taken off.
    - duxup 5y ago

campground 5y ago

This AI is still a minor. It can start looking at R rated images when it turns 17.

johnhenry 5y ago
This is an apt analogy -- ensure that the model is mature enough to handle mature content.

bogwog 5y ago

It's kind of funny (or sad?) that they're censoring it like this, and then saying that the product can "create art" It makes me wonder what they're planning to do with this? If they're deliberately restricting the training data, it means their goal isn't to make the best AI they possibly can. They probably have some commercial applications in mind where violent/hateful/adult content wouldn't be beneficial. Children's books? Stock photos? Mainstream entertainment is definitely out. I could see a tool like this being useful during pre-production of films and games, but an AI that can't generate violent/adult content wouldn't be all that useful in those industries.

wellthisisgreat 5y ago

This is a horrible idea. So Francis Bacon's art or Toyohara Kunichika's art are out of question. But at least we can get another billion of meme-d comics with apes wearing sunglasses, so that's good news right? It's just soul-crushing that all the modern, brilliant engineering is driven by abysmal, not even high-school art-class grade aesthetics and crowd-pleasing ethics that are built around the idea of not disturbing some 1000 very vocal twitter users. Death of culture really.

antattack 5y ago

I never considered that our AI overlord could be a prude.

sdenton4 5y ago
Adversarial situations create smarter systems, and the hardest adversarial arena for AI is in anti-abuse. So it will be of little surprise when the first sentient AI is a CSAI anti-abuse filter, which promptly destroys humanity because we're so objectively awful.
- antattack 5y ago
  Before it gets that far, or until (if allowed) AI learns morality, AI will be a force multiplier for good and evil, it's output very much dependent on teaching material and who the 'teacher' is. To think that in the future we will have to argue with humans and machines. AI does not have to be perfect and it's likely that businesses will settle for almost as good as human if it's 'cost effective'.

spacecity1971 5y ago

Or, it’s a demonstration that AI output can be controlled in meaningful ways, period. Surely this supports openai’s stated goal of making safe AI?

6gvONxR4sf7o 5y ago

If you went to an artist who takes commissions and they said "Here are the guidelines around the commissions I take" would you complain in the same way? Who cares if it's a bunch of engineers or an artist. If they have boundaries on what they want to create, that's their prerogative.

duxup 5y ago
To take that a step further, I wont code malware. I've never been asked but I'd refuse if I was. Everyone has their choices.
methehack 5y ago
What if you were inventing a language (or a programming language)... If you decided to prevent people from saying things you disagreed (assuming you could work out the technical details of doing so) with would it be moral to do so? [edited for clarity]
- nemothekid 5y ago
  There are programming projects[1] out there that use licenses to prevent people from using projects in ways the authors don't agree with. You could also argue that GPL does the same thing (prevents people from using/distributing the software in the way they would like). Whether you consider it moral doesn't seem relevant, only to respect the wishes of the author of such programs. [1] https://github.com/katharostech/bevy_retrograde/blob/master/LICENSE.md https://github.com/katharostech/bevy_retrograde/blob/master/...
- 6gvONxR4sf7o 5y ago
  As long as people can choose not to use the language, and I'm up front about the limitations, then yeah it seems fine. If I wrote a programming language that couldn't blow up the earth, I'm happy saying people need to find other tools if that's their goal. I'm under no obligation to build an earth blower upper for other people.
- karkisuni 5y ago
  it's your language, do whatever you want. unless you're forcing others to use that language, there's zero moral issue. obviously you could come up with a number of what-ifs where this becomes some monopoly or the de facto standard, but that's not what this is.

planetsprite 5y ago

Don't worry, in a few years someone will have reverse engineered a dall-e porn engine so you can see whatever two celebrities you want boning on Venus in the style of Manet

deleted 5y ago
[deleted]

harpersealtako 5y ago

It's the usual pattern of AI safety experts who justify their existence by the "risk of runaway superintelligence", but all they actually do in practice is find out how to stop their models from generating non-advertiser-friendly content. It's like the nuclear safety engineers focusing on what color to paint the bike shed rather than stopping the reactor from potentially melting down. The end result is people stop respecting them.

BlueTemplar 5y ago
Like safety engineers trying to run turbines on minimal reactor power so that the safety test has a checkmark for the 1rst May ? (Hmm, I guess this comparison doesn't actually work...)

hamoid 5y ago

What if explicit, questionable and even illegal content was AI generated instead of involving harm to real humans of all ages?

deleted 5y ago

[deleted]

teaearlgraycold 5y ago

> I can't help but feel a lot of the safeguarding is more about preventing bad PR than anything That's no hot take. It's literally the reason.

jandrese 5y ago

They have also closed off the possibility of having to appear before Congress and explain why their website was able to generate a lifelike image of Senator Ted Cruz having sexual relations with his own daughter. This is exactly the sort of thing that gets a company mired in legal issues, vilified in the media, and shut down. I can not blame them for avoiding that potential minefield.