13 ms·

X faces global investigations for deepfake porn of women and minors

109 points by ryandrake 10mo ago

llmslave2 10mo ago
X needs to be held liable in a criminal court for this. They should not be allowed to hide behind an AI and claim they are not at fault or whatever. They provide a service where users can request images, and the service generated illegal imagery. I will die on the hill that engineers and execs should be heavily fined or going to jail for this.
- devilsdata 10mo ago
  Hear, hear. There should always be steep consequences for taking advantage of the vulnerable.
- ares623 10mo ago
  I know Musk will be untouchable but if X has offices around the world can the people there be held accountable?
  - ronsor 10mo ago
    X will simply close those offices, as they've done in the past.
    - thedevilslawyer 10mo ago
      This is not a bad outcome.
- TrainedMonkey 10mo ago
  While we are at it, let's also get all the weapon manufacturers. How dare they hide behind "we are just making a tool" and "we are not responsible for what our users do". And don't get me started on people using cars to cause harm to others, why is nobody going after big auto?
  - arcatech 10mo ago
    Flattening the discussion to “everything is actually exactly the same” isn’t helpful. This situation is not the same as a person driving a vehicle.
  - llmslave2 10mo ago
    No, because Grok or ChatGPT etc are not tools, they are services. It doesn't matter if Grok uses an LLM or a bunch of people using Photoshop, if they are outputting illegal content the consequences should be the same. People don't go after big auto because they sell cars that other people drive. If they suddenly started offering a service where they would drive people around in their cars, and they started crashing into other people, of course they would be responsible for that.
  - janice1999 10mo ago
    What an unserious response. Gun manufacturer accountability (or lack thereof) is a complex topic and one with ongoing lawsuits and evolving legal arguments. (In the USA see the Bush-era NRA-backed Protection of Lawful Commerce in Arms Act in 2005 and recent Smith & Wesson Brands, Inc. v. Mexico arguments). Personally if a gun manufacturer markets a gun with the primary feature being finger print resistant (yes, that's real) and being easy to carry concealed, I think lawmakers should investigate. Likewise f someone makes a big CSAM generator button and puts it in front of millions of users, it also deserves legal attention.
    - TrainedMonkey 10mo ago
      It was an unserious response to an absolutist statement. I pointing out where such absolutism would lead if applied to other areas. I don't think fingerprint scanner on guns will be effective as it tracks ownership and not legality of usage. However, a number of modern vehicles do have capabilities to perform autonomous actions, including overriding user input.
      - llmslave2 10mo ago
        It's not an absolutist statement and it doesn't lead to any of the insane conclusions you came up with. Holding the people running a model liable for what it generates has nothing to do with making tool makers liable.
- s5300 10mo ago
  [dead]
- dmitrygr 10mo ago
  > service generated illegal imagery For my own curiosity, could you please elaborate which law was broken? And if you do find one, elaborate how is it not unconstitutional, given that “put X into a bikini” is constitutionally protected speech, and the output fails the Roth test for obscenity and is thus protected just the same. Would not every other law claiming to be relevant be null and void w.r.t. stoping this in this particular case? SCOTUS was quite clear in Roth v. United States. The later Miller case is also of no help, since nowadays appearing publicly in a bikini is considered quite normal and not at all obscene or sexual. I get it. If this happened to a member of my family, I'd be pissed too. But, as logical people is it not our job to NOT simply leap from "I'm very angry" to "It must be illegal", but to be calm and rational.
  - labrador 10mo ago
    It's illegal in California.
    - dmitrygr 10mo ago
      > It's illegal in California. 1. What specifically is? Posting twitter comments? Prompting an ML model? Generating images based on existing images? 2. Even if so, states are not allowed to make unconstitutional laws. 1stA covers this, as far as I can tell. Do you read Roth or Miller differently?
      - labrador 10mo ago
        "is posting nudes of women without their consent legal in california" https://www.google.com/search?q=is+posting+nudes+of+women+without+their+consent+legal+in+california https://www.google.com/search?q=is+posting+nudes+of+women+wi...
        xigoi 10mo ago
        Posting such images is illegal, i.e. it’s the user who has violated the law.
afavour 10mo ago
[flagged]
- FloorEgg 10mo ago
  Maybe start by reviewing the HN guidelines and asking yourself whether or not this is on-topic for HN.
  - square_usual 10mo ago
    How is this not on topic for HN?
    - groby_b 10mo ago
      Given the guideline is "anything that gratifies one's intellectual curiosity." one could certainly make the case it's not a topic that's fit. "Large company does horrible things, occasionally an investigation happens" is not satisfying curiosity on an intellectual level. What, specifically, are we learning? Yes, we can debate if it's relevant to the audience anyways, or if it's healthy for HN to have that guideline, but those are different criteria & topics.
      - danudey 10mo ago
        > What, specifically, are we learning? That some countries are willing to hold tech companies accountable (or claim their intent to hold tech companies accountable) for breaking the laws. In turn, this could lead to court cases deciding if companies can hide behind "It wasn't us, our AI did it!" or if they have to take responsibility for the thing they made.
      - frm88 10mo ago
        "Large company does horrible things, occasionally an investigation happens" is not satisfying curiosity on an intellectual level. What, specifically, are we learning? By your metric most of the topics concerning Google or Microsoft in the past months should not have staid on the front page, let alone most of the American politics topics. And yet, here we are: anything critical of any of Musk's companies gets flagged, American politics stay. @dang: how does this flagging thingy work? How many user flags are necessary until a topic appears as [flagged]? Thank you in advance for an explanation.
huflungdung 10mo ago
[dead]
llmslave2 10mo ago
[flagged]
labrador 10mo ago
This shouldn't be flagged because this is a very interesting AI problem. Grok was trained on X content which contains a lot of porn. Other image generation models aren't trained on porn so they don't know how to produce it. It appears to be very difficult to stop Grok from making porn since it has been trained on it. Is the ony workable solution to not train Grok 5 on porn?
- janice1999 10mo ago
  They would never do that. They have an anime girlfriend app. They've picked their target audience, let's if it was worth it.
queenkjuul 10mo ago
Oh wow somebody decided news about an international tech giant is suddenly irrelevant on a site about tech news, crazy how that happens with one particular category of tech news...
soraminazuki 10mo ago
It's hardly the first time, but we have yet again confirmed that the people who say they flag these posts because it's not related to tech are outright lying.
- Jordan-117 10mo ago
  The real front page: https://news.ycombinator.com/active https://news.ycombinator.com/active
- FloorEgg 10mo ago
  As a well intentioned user (with no stake whatsoever in Grok or parties involved) who originally flagged this but then decided to remove my flag, here was my original reasoning... In the HN guidelines there are two category definitions, on-topic and off-topic. I don't think there is an intellectually honest argument to make that this submission is not on-topic, but there is an argument to make that it is off-topic. - it's related to a celebrity - it's related to crime - it's likely to appear on TV news I'd also add that it's likely to spur strong emotions, political debate and any attempt by people to try and understand the actual legal nuances (e.g. how does each legal jurisdiction decide who is accountable, the platform, the user, how is the platform held accountable exactly, etc.) is likely to be labelled a sea lion and down voted by someone who just says "this is obviously illegal". In other words this topic is not conducive to intellectually curious discussion, which I think is the spirit of HN guidelines. In other words there is a lot of noise between people who curiously engage in this topic, and the people who are offended by Grok and the people associated with it, and anyone who isn't also outraged and grabbing their pitchforks. You are making an assumption that anyone who would label this as off topic is outright lying about why, and it may be that some people are, but I am not. I do think there is a benefit of the doubt interpretation to be made, which you may think is naive, but seems less naive for anyone who genuinely values HN as a sanctuary from the inflammatory dialog found in most other social media. Further, when I attempted to converse with others about why this could be considered off-topic (another comment thread under this submission that you replied to), my character was repeatedly attacked, and many unfounded assumptions about my motives were made. These were clear signs to me that people were struggling to engage curiously.
kyleee 10mo ago
Where are the examples? How do we know this story is true?

109 points by ryandrake 10mo ago

X needs to be held liable in a criminal court for this. They should not be allowed to hide behind an AI and claim they are not at fault or whatever. They provide a service where users can request images, and the service generated illegal imagery. I will die on the hill that engineers and execs should be heavily fined or going to jail for this.

devilsdata 10mo ago
Hear, hear. There should always be steep consequences for taking advantage of the vulnerable.
ares623 10mo ago
I know Musk will be untouchable but if X has offices around the world can the people there be held accountable?
- ronsor 10mo ago
  X will simply close those offices, as they've done in the past.
  - thedevilslawyer 10mo ago
    This is not a bad outcome.
TrainedMonkey 10mo ago
While we are at it, let's also get all the weapon manufacturers. How dare they hide behind "we are just making a tool" and "we are not responsible for what our users do". And don't get me started on people using cars to cause harm to others, why is nobody going after big auto?
- arcatech 10mo ago
  Flattening the discussion to “everything is actually exactly the same” isn’t helpful. This situation is not the same as a person driving a vehicle.
- llmslave2 10mo ago
  No, because Grok or ChatGPT etc are not tools, they are services. It doesn't matter if Grok uses an LLM or a bunch of people using Photoshop, if they are outputting illegal content the consequences should be the same. People don't go after big auto because they sell cars that other people drive. If they suddenly started offering a service where they would drive people around in their cars, and they started crashing into other people, of course they would be responsible for that.
- janice1999 10mo ago
  What an unserious response. Gun manufacturer accountability (or lack thereof) is a complex topic and one with ongoing lawsuits and evolving legal arguments. (In the USA see the Bush-era NRA-backed Protection of Lawful Commerce in Arms Act in 2005 and recent Smith & Wesson Brands, Inc. v. Mexico arguments). Personally if a gun manufacturer markets a gun with the primary feature being finger print resistant (yes, that's real) and being easy to carry concealed, I think lawmakers should investigate. Likewise f someone makes a big CSAM generator button and puts it in front of millions of users, it also deserves legal attention.
  - TrainedMonkey 10mo ago
    It was an unserious response to an absolutist statement. I pointing out where such absolutism would lead if applied to other areas. I don't think fingerprint scanner on guns will be effective as it tracks ownership and not legality of usage. However, a number of modern vehicles do have capabilities to perform autonomous actions, including overriding user input.
    - llmslave2 10mo ago
      It's not an absolutist statement and it doesn't lead to any of the insane conclusions you came up with. Holding the people running a model liable for what it generates has nothing to do with making tool makers liable.
s5300 10mo ago
[dead]
dmitrygr 10mo ago
> service generated illegal imagery For my own curiosity, could you please elaborate which law was broken? And if you do find one, elaborate how is it not unconstitutional, given that “put X into a bikini” is constitutionally protected speech, and the output fails the Roth test for obscenity and is thus protected just the same. Would not every other law claiming to be relevant be null and void w.r.t. stoping this in this particular case? SCOTUS was quite clear in Roth v. United States. The later Miller case is also of no help, since nowadays appearing publicly in a bikini is considered quite normal and not at all obscene or sexual. I get it. If this happened to a member of my family, I'd be pissed too. But, as logical people is it not our job to NOT simply leap from "I'm very angry" to "It must be illegal", but to be calm and rational.
- labrador 10mo ago
  It's illegal in California.
  - dmitrygr 10mo ago
    > It's illegal in California. 1. What specifically is? Posting twitter comments? Prompting an ML model? Generating images based on existing images? 2. Even if so, states are not allowed to make unconstitutional laws. 1stA covers this, as far as I can tell. Do you read Roth or Miller differently?
    - labrador 10mo ago
      "is posting nudes of women without their consent legal in california" https://www.google.com/search?q=is+posting+nudes+of+women+without+their+consent+legal+in+california https://www.google.com/search?q=is+posting+nudes+of+women+wi...
      - xigoi 10mo ago
        Posting such images is illegal, i.e. it’s the user who has violated the law.

afavour 10mo ago

[flagged]

FloorEgg 10mo ago
Maybe start by reviewing the HN guidelines and asking yourself whether or not this is on-topic for HN.
- square_usual 10mo ago
  How is this not on topic for HN?
  - groby_b 10mo ago
    Given the guideline is "anything that gratifies one's intellectual curiosity." one could certainly make the case it's not a topic that's fit. "Large company does horrible things, occasionally an investigation happens" is not satisfying curiosity on an intellectual level. What, specifically, are we learning? Yes, we can debate if it's relevant to the audience anyways, or if it's healthy for HN to have that guideline, but those are different criteria & topics.
    - danudey 10mo ago
      > What, specifically, are we learning? That some countries are willing to hold tech companies accountable (or claim their intent to hold tech companies accountable) for breaking the laws. In turn, this could lead to court cases deciding if companies can hide behind "It wasn't us, our AI did it!" or if they have to take responsibility for the thing they made.
    - frm88 10mo ago
      "Large company does horrible things, occasionally an investigation happens" is not satisfying curiosity on an intellectual level. What, specifically, are we learning? By your metric most of the topics concerning Google or Microsoft in the past months should not have staid on the front page, let alone most of the American politics topics. And yet, here we are: anything critical of any of Musk's companies gets flagged, American politics stay. @dang: how does this flagging thingy work? How many user flags are necessary until a topic appears as [flagged]? Thank you in advance for an explanation.

huflungdung 10mo ago

[dead]

llmslave2 10mo ago

[flagged]

labrador 10mo ago

This shouldn't be flagged because this is a very interesting AI problem. Grok was trained on X content which contains a lot of porn. Other image generation models aren't trained on porn so they don't know how to produce it. It appears to be very difficult to stop Grok from making porn since it has been trained on it. Is the ony workable solution to not train Grok 5 on porn?

janice1999 10mo ago
They would never do that. They have an anime girlfriend app. They've picked their target audience, let's if it was worth it.

queenkjuul 10mo ago

Oh wow somebody decided news about an international tech giant is suddenly irrelevant on a site about tech news, crazy how that happens with one particular category of tech news...

soraminazuki 10mo ago

It's hardly the first time, but we have yet again confirmed that the people who say they flag these posts because it's not related to tech are outright lying.

Jordan-117 10mo ago
The real front page: https://news.ycombinator.com/active https://news.ycombinator.com/active
FloorEgg 10mo ago
As a well intentioned user (with no stake whatsoever in Grok or parties involved) who originally flagged this but then decided to remove my flag, here was my original reasoning... In the HN guidelines there are two category definitions, on-topic and off-topic. I don't think there is an intellectually honest argument to make that this submission is not on-topic, but there is an argument to make that it is off-topic. - it's related to a celebrity - it's related to crime - it's likely to appear on TV news I'd also add that it's likely to spur strong emotions, political debate and any attempt by people to try and understand the actual legal nuances (e.g. how does each legal jurisdiction decide who is accountable, the platform, the user, how is the platform held accountable exactly, etc.) is likely to be labelled a sea lion and down voted by someone who just says "this is obviously illegal". In other words this topic is not conducive to intellectually curious discussion, which I think is the spirit of HN guidelines. In other words there is a lot of noise between people who curiously engage in this topic, and the people who are offended by Grok and the people associated with it, and anyone who isn't also outraged and grabbing their pitchforks. You are making an assumption that anyone who would label this as off topic is outright lying about why, and it may be that some people are, but I am not. I do think there is a benefit of the doubt interpretation to be made, which you may think is naive, but seems less naive for anyone who genuinely values HN as a sanctuary from the inflammatory dialog found in most other social media. Further, when I attempted to converse with others about why this could be considered off-topic (another comment thread under this submission that you replied to), my character was repeatedly attacked, and many unfounded assumptions about my motives were made. These were clear signs to me that people were struggling to engage curiously.

kyleee 10mo ago

Where are the examples? How do we know this story is true?