3 ms·

Taken literally this comment suggests that OpenAI et al actually want racist 4chan rants in ChatGPT's training data, or illegal child pornography in DALL-E's tr

Taken literally this comment suggests that OpenAI et al actually want racist 4chan rants in ChatGPT's training data, or illegal child pornography in DALL-E's training data. Obviously that's not the case: the problem is that it's impossible to effectively audit the amount of data required to make generative AI work. If Big Tech can't even catch pornography or explicit uses of the n-word, there's no way they'll be able to filter subtle LLM / art generator hallucinations.

jsheard 2y ago
Going one level deeper, even AI-generated training data which doesn't contain hallucinations or other obvious errors could still negatively affect the performance of a new model. GenAI models have a tendency to repeat the same cliches, in sentence structure or the composition of images or whatever else, so even if their output is "fine" in the sense that it's factually correct and everyone has the right number of fingers, a model trained on it will still internalize those cliches and make them even more likely to show up again. That's even harder to audit out because it would only become apparent in aggregate when you realise your new model has learned to interpret "fantasy artwork" as "fantasy artwork in the generic Midjourney RLHF style" since it was trained on so many examples of that.
- wruza 2y ago
  That's really just a feature of the default system prompt, although ubiquitous. You can character-prompt local LLMs as you like, but no one seems to do that.
  - CuriouslyC 2y ago
    ChatGPT doesn't take character prompts very well, that's why people don't know. Claude on the other hand is a champ with characters.
simonw 2y ago
Has it been demonstrated that there is illegal CSAM in the DALL-E training?
- candiodari 2y ago
  Why would you need to? Obviously generalizing over pictures of humans and pictures of porn will enable CSAM generation ... Plus a LOT of things have been declared illegal pornographic material in various places: all porn (China), drawn CSAM, any kind of drawn porn with unclear ages, ...
  - simonw 2y ago
    > "Why would you need to?" If someone is going to suggest that there is "illegal child pornography in DALL-E's training data" I think it needs to be backed up - that's a pretty big accusation.
- jsheard 2y ago
  I don't think it has, if only because OpenAI aren't transparent about where they get their image training data from, but the most widely used public image generation dataset was found to contain CSAM: https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford https://www.theverge.com/2023/12/20/24009418/generative-ai-i...
- nicklecompte 2y ago
  It hasn't been demonstrated in DALL-E, but this was a problem in older OpenAI text-video models, and it's in plenty of CommonCrawl/etc data. There is anecdata from contractors suggesting that DALL-E generated CSAM imagery during RLHF, which is circumstantial evidence that similar images are somewhere in the training data. I think it's fair to say we can't rely on OpenAI's technical competence to remove this stuff from their training data, despite claims to the contrary w.r.t DALL-E 2. (Likewise it seems Copilot Designer churns out CSAM like candy.) Your attitude is naive and competely at odds with (honest) AI safety research. The rational thing for users is to assume DALL-E 2 is contaminated by CSAM unless OpenAI can publicly demonstrate otherwise. Their leadership is too stupid and dishonest for people to take their word. https://ieeexplore.ieee.org/abstract/document/9423393 https://ieeexplore.ieee.org/abstract/document/9423393 https://arxiv.org/pdf/2110.01963.pdf?trk=public_post_comment-text https://arxiv.org/pdf/2110.01963.pdf?trk=public_post_comment... https://www.theguardian.com/technology/2023/aug/02/ai-chatbot-training-human-toll-content-moderator-meta-openai https://www.theguardian.com/technology/2023/aug/02/ai-chatbo...
CuriouslyC 2y ago
Not so much that, as OpenAI has finite resources and is allocating them to areas that are more valuable.

jsheard 2y ago

Going one level deeper, even AI-generated training data which doesn't contain hallucinations or other obvious errors could still negatively affect the performance of a new model. GenAI models have a tendency to repeat the same cliches, in sentence structure or the composition of images or whatever else, so even if their output is "fine" in the sense that it's factually correct and everyone has the right number of fingers, a model trained on it will still internalize those cliches and make them even more likely to show up again. That's even harder to audit out because it would only become apparent in aggregate when you realise your new model has learned to interpret "fantasy artwork" as "fantasy artwork in the generic Midjourney RLHF style" since it was trained on so many examples of that.

wruza 2y ago
That's really just a feature of the default system prompt, although ubiquitous. You can character-prompt local LLMs as you like, but no one seems to do that.
- CuriouslyC 2y ago
  ChatGPT doesn't take character prompts very well, that's why people don't know. Claude on the other hand is a champ with characters.

simonw 2y ago

Has it been demonstrated that there is illegal CSAM in the DALL-E training?

candiodari 2y ago
Why would you need to? Obviously generalizing over pictures of humans and pictures of porn will enable CSAM generation ... Plus a LOT of things have been declared illegal pornographic material in various places: all porn (China), drawn CSAM, any kind of drawn porn with unclear ages, ...
- simonw 2y ago
  > "Why would you need to?" If someone is going to suggest that there is "illegal child pornography in DALL-E's training data" I think it needs to be backed up - that's a pretty big accusation.
jsheard 2y ago
I don't think it has, if only because OpenAI aren't transparent about where they get their image training data from, but the most widely used public image generation dataset was found to contain CSAM: https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford https://www.theverge.com/2023/12/20/24009418/generative-ai-i...
nicklecompte 2y ago
It hasn't been demonstrated in DALL-E, but this was a problem in older OpenAI text-video models, and it's in plenty of CommonCrawl/etc data. There is anecdata from contractors suggesting that DALL-E generated CSAM imagery during RLHF, which is circumstantial evidence that similar images are somewhere in the training data. I think it's fair to say we can't rely on OpenAI's technical competence to remove this stuff from their training data, despite claims to the contrary w.r.t DALL-E 2. (Likewise it seems Copilot Designer churns out CSAM like candy.) Your attitude is naive and competely at odds with (honest) AI safety research. The rational thing for users is to assume DALL-E 2 is contaminated by CSAM unless OpenAI can publicly demonstrate otherwise. Their leadership is too stupid and dishonest for people to take their word. https://ieeexplore.ieee.org/abstract/document/9423393 https://ieeexplore.ieee.org/abstract/document/9423393 https://arxiv.org/pdf/2110.01963.pdf?trk=public_post_comment-text https://arxiv.org/pdf/2110.01963.pdf?trk=public_post_comment... https://www.theguardian.com/technology/2023/aug/02/ai-chatbot-training-human-toll-content-moderator-meta-openai https://www.theguardian.com/technology/2023/aug/02/ai-chatbo...

CuriouslyC 2y ago

Not so much that, as OpenAI has finite resources and is allocating them to areas that are more valuable.