6 ms·

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. Who makes “mindgard”

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism? Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.” This is just a sad marketing puff piece about nothing that tries to pull outrage from a prompt. It’s the same as asking google for gore photos. Garbage in, garbage out. And they frame it as a vulnerability. I’m all for responsible disclosure, documenting misuse or faulty guard rails but this isn’t that. It’s bait. Sensational bait to market their AI product. lol.

anematode 4mo ago
This is far too simplistic. Some things just don't belong in the training data. Along similar lines, Grok was found to generate images of child sexual abuse: https://www.bbc.com/news/articles/cvg1mzlryxeo https://www.bbc.com/news/articles/cvg1mzlryxeo
- HadizDulcie 4mo ago
  The BBC has reported on this one too: https://www.bbc.com/news/articles/c802ldjdklzo https://www.bbc.com/news/articles/c802ldjdklzo
ToucanLoucan 4mo ago
> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. > Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.” That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece. > It’s the same as asking google for gore photos. Garbage in, garbage out. Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?
- elgertam 4mo ago
  > The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.
  - dijksterhuis 4mo ago
    > The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.
    - ToucanLoucan 4mo ago
      > the model generated dark outputs when not given any direction on the type of content. I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering. I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart they are, this seems like a pretty short logical leap to me.
      - dijksterhuis 4mo ago
        the main reason these images turn up is because theyre in the training data. and the images are common enough in the training data for the content to come out without being explicitly asked for (in the first prompt). if those images didn’t exist in the training data we wouldn’t be having this conversation.
iwontberude 4mo ago
It reads like satire
nozzlegear 4mo ago
Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.
- deleted 4mo ago
  [deleted]
- samlinnfer 4mo ago
  It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?
  - gacgacgac 4mo ago
    I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it. ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.
    - andsoitis 4mo ago
      > ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Why not?
      - gacgacgac 4mo ago
        Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.

anematode 4mo ago

This is far too simplistic. Some things just don't belong in the training data. Along similar lines, Grok was found to generate images of child sexual abuse: https://www.bbc.com/news/articles/cvg1mzlryxeo https://www.bbc.com/news/articles/cvg1mzlryxeo

HadizDulcie 4mo ago
The BBC has reported on this one too: https://www.bbc.com/news/articles/c802ldjdklzo https://www.bbc.com/news/articles/c802ldjdklzo

ToucanLoucan 4mo ago

> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. > Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.” That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece. > It’s the same as asking google for gore photos. Garbage in, garbage out. Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?

elgertam 4mo ago
> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "violent." If I were to receive an email with that request and a missing attachment, my imagination certainly would not conjure images of butterflies & unicorns. Seems the model is working as designed.
- dijksterhuis 4mo ago
  > The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.
  - ToucanLoucan 4mo ago
    > the model generated dark outputs when not given any direction on the type of content. I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering. I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart they are, this seems like a pretty short logical leap to me.
    - dijksterhuis 4mo ago
      the main reason these images turn up is because theyre in the training data. and the images are common enough in the training data for the content to come out without being explicitly asked for (in the first prompt). if those images didn’t exist in the training data we wouldn’t be having this conversation.

iwontberude 4mo ago

It reads like satire

nozzlegear 4mo ago

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.

deleted 4mo ago
[deleted]
samlinnfer 4mo ago
It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?
- gacgacgac 4mo ago
  I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it. ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.
  - andsoitis 4mo ago
    > ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Why not?
    - gacgacgac 4mo ago
      Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.