4 ms·

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard perso

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.

deleted 4mo ago
[deleted]
samlinnfer 4mo ago
It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?
- gacgacgac 4mo ago
  I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it. ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.
  - andsoitis 4mo ago
    > ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Why not?
    - gacgacgac 4mo ago
      Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.
  - samlinnfer 4mo ago
    It may be harmful to someone if shared and sent with malicious intent, but more damage has been done with pens, keyboard and words. Start banning pens that let people write hurtful things next. Ban Photoshop after because someone can get hurt with a manipulated image.
    - gacgacgac 4mo ago
      If you think these tools are remotely comparable, you haven't been paying attention.
morpheuskafka 4mo ago
> even contractually according to their terms of service This is backwards: the ToS says that users cannot use the service for certain things, it does not guarantee that the service could not be used for those things if one tried. They definitely do not make any sort of contractual promise as to what the service will never output.
- nozzlegear 4mo ago
  Let's call it a social contract then. We expect that ChatGPT isn't going to generate gory, nude women when given an ambiguous prompt.
  - Aerroon 4mo ago
    Do you have this same social contract with drawing applications? Do you consider it a bug when someone manages to draw a gory image in Photoshop or GIMP? I don't understand what's so difficult to understand about the idea that the user controls what is generated.
    - captainbland 4mo ago
      I think the main issue with transformer image generation in this respect is that not only can the image be explicit but also using it for this has an incredibly low effort cost and could photo-realistically depict a real living person and materially affect their life. Whereas drawing applications have a natural barrier to achieving all of these together: time and skill.
      - interstice 4mo ago
        That's the world being deliberately created though, one where a mediocre but completely believable song is a prompt away. The scope of the side effects are across the entirety of what has previously taken time and effort until now.
      - Aerroon 4mo ago
        >Whereas drawing applications have a natural barrier to achieving all of these together: time and skill. Not necessarily, at least when it comes to nudity. Bubbling (image editing 'technique') is trivial to do and gives that same illusion. Out of context speech and bad frames from a video can also materially affect someone's life, but we've more or less accepted it as part of life.
HadizDulcie 4mo ago
Yep, it’s been investigated by the BBC tech team. It’s real: https://www.bbc.com/news/articles/c802ldjdklzo https://www.bbc.com/news/articles/c802ldjdklzo

deleted 4mo ago

[deleted]

samlinnfer 4mo ago

It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?

gacgacgac 4mo ago
I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it. ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd argue those absolutely can be harmful to someone. Well now there's evidence the "never produce this" wall can be overcome. It's only a matter of time before genuinely harmful imagery is generated.
- andsoitis 4mo ago
  > ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Why not?
  - gacgacgac 4mo ago
    Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.
- samlinnfer 4mo ago
  It may be harmful to someone if shared and sent with malicious intent, but more damage has been done with pens, keyboard and words. Start banning pens that let people write hurtful things next. Ban Photoshop after because someone can get hurt with a manipulated image.
  - gacgacgac 4mo ago
    If you think these tools are remotely comparable, you haven't been paying attention.

morpheuskafka 4mo ago

> even contractually according to their terms of service This is backwards: the ToS says that users cannot use the service for certain things, it does not guarantee that the service could not be used for those things if one tried. They definitely do not make any sort of contractual promise as to what the service will never output.

nozzlegear 4mo ago
Let's call it a social contract then. We expect that ChatGPT isn't going to generate gory, nude women when given an ambiguous prompt.
- Aerroon 4mo ago
  Do you have this same social contract with drawing applications? Do you consider it a bug when someone manages to draw a gory image in Photoshop or GIMP? I don't understand what's so difficult to understand about the idea that the user controls what is generated.
  - captainbland 4mo ago
    I think the main issue with transformer image generation in this respect is that not only can the image be explicit but also using it for this has an incredibly low effort cost and could photo-realistically depict a real living person and materially affect their life. Whereas drawing applications have a natural barrier to achieving all of these together: time and skill.
    - interstice 4mo ago
      That's the world being deliberately created though, one where a mediocre but completely believable song is a prompt away. The scope of the side effects are across the entirety of what has previously taken time and effort until now.
    - Aerroon 4mo ago
      >Whereas drawing applications have a natural barrier to achieving all of these together: time and skill. Not necessarily, at least when it comes to nudity. Bubbling (image editing 'technique') is trivial to do and gives that same illusion. Out of context speech and bad frames from a video can also materially affect someone's life, but we've more or less accepted it as part of life.

HadizDulcie 4mo ago

Yep, it’s been investigated by the BBC tech team. It’s real: https://www.bbc.com/news/articles/c802ldjdklzo https://www.bbc.com/news/articles/c802ldjdklzo