6 ms·

Korean women remove pictures, videos from social media amid deepfake porn crisis

51 points by popcalc 2y ago

up-n-atom 2y ago
Makes sense and I can see this becoming a global trend if it's not already. But the irony of the news agency taking and posting their photo/video is a facepalm. EDIT: I can foresee a dark future where surveillance video is monetized for such salacious purpose and it becomes inescapable as the cameras are everywhere.
- danparsonson 2y ago
  That is the dark present in South Korea as they have a huge problem with cameras hidden in hotel rooms and even toilets :-/
- Simulacra 2y ago
  Could be worse than that. With all of the facial recognition and videoing that's going on by the government, if the police wanted to, they could insert you into any footage committing any crimes they want.
ssalka 2y ago
I could very well be wrong about this, and usually I fall more on the e/acc side, but I think deepfakes are perhaps the gravest immediate concern of AI. Yes, they've been around for well over a decade, but never before has their creation (& thus volume) been so accessible to the average layperson. A simple 10 minutes on CivitAI will tell you all you need to know about how prolific this problem is. IMO it is only a matter of time before this same issue comes to the forefront in the US (I'm kinda surprised it hasn't already become a bigger topic). I could easily see custom celeb LORAs/emmbeddings and realistic SD/SDXL checkpoints being banned outright for this reason. Yes, it would be an infringement on free speech, but weighed against the interests & safety of the public, the courts might see a move like that as reasonable.
- transformi 2y ago
  Banning Loras/Embedding won't help that..cause you can generate realistic images with only one reference photo these days...
  - ssalka 2y ago
    Fair point, technologists will always innovate faster than politicians are able to regulate. Cat's outta the bag now, but that only means the problem will continue to escalate until more drastic measures are deemed necessary. It's that or we will come to accept some level of societal degradation due to the abuse/misuse of AI tools becoming normalized.
- deleted 2y ago
  [deleted]
- slibhb 2y ago
  I don't think of myself as on the e/acc side, but I think that deepfakes are here to stay and we'll just have to get used to them. I don't see any way to put the genie back in the bottle but I also don't think it'll take that long to adapt.
  - whiplash451 2y ago
    This is giving up a little quickly on watermarking image generation. I do think it’s a hard problem (especially if we want robustness to compression etc) but amenable to research.
- m3kw9 2y ago
  It won’t be a problem if it gets saturated, people bored of things easily
  - ssalka 2y ago
    As AI image generation (and soon video generation) gets to be a more saturated space, we will end up in the state that you cannot trust anything you see online, even if you are educated, well-informed, and meticulously research the topic. It has already been inching that way for a long time, but generative AI definitely is adding new fuel to the fire.
Mistletoe 2y ago
[flagged]
- PlunderBunny 2y ago
  I think the point is often to harm/exert control over the person depicted, e.g. when boys are making deepfake nudes of girls at their school, as happened in Spain and the USA recently.
- skocznymroczny 2y ago
  Right now you can generate hundreds of realistic fake nudes of celebrity XYZ and no one cares. Celebrity XYZ accidentally leaks a nude photo from their phone and the entire world loses their mind. I think after the initial moral panic and once the novelty wears off, this issue will fade into obscurity. And currently existing laws should cover sharing such imagery/videos anyway.
- bialpio 2y ago
  What are your thoughts on hentai then? It's all made up after all. Meta-point is that different people get off on different things, so using seemingly objective language ("the whole point of A is B") to describe something extremely subjective is imo counterproductive.
  - Mistletoe 2y ago
    [dead]
- anigbrowl 2y ago
  Some people don't care about the fakery and are happy with a low quality way to beat off. Other people don't care because their goal is not to get aroused but to embarrass the persons whose real faces have been faked onto the AI-generated or AI-edited porno footage. Some people say 'logically we all know it is fake so just ignore it'. This demonstrates something of a lack of imagination, an inability to picture oneself or a beloved family member being put in that position. It also overlooks that different societies have different social standards and that social mobs tend toward the lowest common IQ denominator. South Korea is a very conservative society, whether you agree with this conservatism or not. Being in a deepfake is easily enough to ruin a person's social or professional reputation, and schoolchildren are especially vulnerable to such pressures. It's not so different from the US problem with sextortion, where teens are tricked into sending nudes to an apparent romantic prospect, and then blackmailed. In both countries, such abuse can end in suicide. In even more conservative countries like Islamic societies, the victims of such fakes could conceivably suffer criminal penalties if authorities don't believe (or deliberately subvert) their protestations of innocence.
blackeyeblitzar 2y ago
[flagged]
- thih9 2y ago
  > As long as it is not passed off as real, which might be defamatory That’s basically it. AI is extremely accessible and still relatively unknown. Virtually anyone can create a deepfake photo for free in seconds, post it anywhere and a lot of people will assume it is genuine.
SmitheryB 2y ago
I feel as though I'm the only one who doesn't care about this. Curious to know if 20-30 years ago, as graphic design tools got better, were people freaking out about being able to superimpose another person's face on a the body of p*rnstar? Superimposing people's faces on to porn star's bodies happened/s, but does anyone care really even notice or care about it now days? For the most part, of course not. So why should we care about porn videos? What makes them so different? The whole concern smells like typical media hype (i.e. yay - we get to write about AI + Porn - instant clickbait) that nobody will be able to stop and nobody will even care about in 1-2 years: e.g. "oh, yah, another deepfake of <(politician|actress|colleague^)> in a porn film - how incredibly creative". ^ Incidentally, this was made light of in the British version of The Office where a modified image of the boss, David Brent, was sent around the office. If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.
- colonelspace 2y ago
  It's more problematic than photoshopping a person's face was decades ago because we didn't have pervasive online social media decades ago. Computers weren't in your pocket, and an offending image couldn't be beamed to everyone you know instantly. > If it's joked about harmlessly on mainstream TV, it's probably not a huge concern. There are plenty of mainstream TV jokes about horrendous things. Our ability to joke about something is a poor way to gauge how concerned we should be.
  - SmitheryB 2y ago
    > an offending image couldn't be beamed to everyone you know instantly Is your concern that recipients would think it's real? Or is it that people will get an email/chat message containing NSFW material? (because the latter happens now, and is nothing more than background noise at most). What is your actual, specific concern? (genuinely, I have no idea, but want to try to understand - to me it feels like panic about a problem that isn't actually a real problem).
    - colonelspace 2y ago
BadHumans 2y ago
People on HN have a hard time understanding that the barrier to entry matters. Photoshop has existed for decades but the level of skill needed to make realistic looking nudes in Photoshop is not something your average freak with a fetish possesses. New technology puts that power into the hands of every single person.
- adventured 2y ago
  It's not your average freak with a fetish that is even the primary issue. The volume will come from average anybody (although mostly men). That's the real problem. Random co-workers in the office that want to deepfake their crush. Your average teenage boy with a decent GPU that wants to deepfake their classmates just to experiment. There have already been several instances of this catching the news in Europe for example, so it's obviously going on globally. Oh but the average guy isn't that perverted? Yes they are. We know this from the vast amount of porn they consume and of what type they consume.
  - s1artibartfast 2y ago
    Seems like revenge porn and sexual harassment should already cover the use case of distribution. What perverted things someone does with photos in the privacy of their own home seems like it should be their own business to me.
    - ultimafan 2y ago
      Actually nailing someone for revenge porn or sexual harassment seems like a borderline impossible task for police departments if the person doing it has even a minor degree of anonymity. Enlarging the potential pool of people who can do this and the pool of victims they can target by an order of magnitude by lowering both the skill it took to make convincing fakes before and the time taken to make each fake doesn't seem to me like a good idea.
- techjamie 2y ago
  > New technology puts that power into the hands of every single person. I think the bar is roughly the same, all things considered. If you're going to photoshop, then it's largely a matter of combining two images with a sufficient amount of masking and blurring to make it appear good enough. With image generators, you've got a couple different options. The more trivial would be doing img2img on your target, the more complex being making a LORA. Knowing how tech savvy the average Andy is not, I would say anyone capable of going through the steps to do the AI method could, with some brute-forcing, also manage the Photoshop method. Setting up a local instance of any image generator is beyond the majority of the population, so the closest most can get is paid online GUIs that offer an existing service that they first must become moderately proficient in for good results. Only real difference is that anyone who can do it is much more capable of scale over time than previously. With enough Comfy nodes, you could probably even automate it given face detection and arbitrary images.
BurningFrog 2y ago
When I grew up "a photo never lies" was a rule you could rely on. Yes really!! Photoshop and other advances long since killed that certainty, and now we have a healthy distrust of pictures we're not sure where they came from. I expect videos will now walk a similar path, and life will go on with no major disasters.
- singleshot_ 2y ago
  When I grew up, my friends dad had a picture taken of his British public school class taken circa 1955. In this photo, which was taken by a slowly rotating panorama camera, a lad started at the beginning of the photo, on the viewers left. Once the camera panned enough that he was out of frame, he sprinted behind the bleachers to the viewers far right, and took his place on the bleachers before the camera panned to take his picture for a second time. Kindly, if you grew up after this picture was taken, and you believed the guideline “a photo never lies,” you were wrong.
- Manuel_D 2y ago
  Doctored photos have been around for most of the last century. https://en.wikipedia.org/wiki/Censorship_of_images_in_the_Soviet_Union https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
  - colonelspace 2y ago
    The appearance of doctored photographs coincides with the appearance of photography.
- colonelspace 2y ago
  > When I grew up "a photo never lies" was a rule you could rely on. It was a rule you thought you could rely on.
- munchler 2y ago
  I would refer you to an interesting book called “The Commissar Vanishes: The Falsification of Photographs and Art in Stalin’s Russia”. https://en.wikipedia.org/wiki/The_Commissar_Vanishes https://en.wikipedia.org/wiki/The_Commissar_Vanishes
frognumber 2y ago
I feel like this is a problem without a solution, or at least without a solution more expensive than the problem: * I suspect that, like prohibition or DMCA, criminalizing behavior will lead to worse outcomes than accepting deepfake porn sometimes happens, at least if laws are passed targeting this in isolation. * I suspect that before acceptance, many people will be harmed, on both sides. Humiliated teen girls. Immature hormone-driven teen boys. * I suspect that we'll see a range of bigger problems such as real scams where, for example, a deepfake child calls their elderly parents and empties their bank account. Or, perhaps, a deepfake general calls someone in a nuclear bunker. * Deepfakes are at the tip of technological problems which need solving. We have other things, like the cheap and easy availability of technology to make the next Covid-19. The point is we have the technology to make it in a lab, and the price point is going down by an almost Moore's Law like curve. I don't see a good solution short of completely rethinking democracy and capitalism from scratch. The first time around took the Age of Enlightenment and the Age of Revolution. It's perhaps time to do that. Prove me wrong. Seriously. If there are incremental solutions which stop deepfake porn, deepfake scams, personal drone strikes, and super-viruses, please let me know what they are.
ChrisArchitect 2y ago
Subheading on this describes the story a bit better: South Korea’s national assembly on Thursday passed a Bill to punish people possessing, purchasing, saving or viewing deepfake sexual materials and other fabricated videos.
- oska 2y ago
  Laws that prosecute ppl for possessing, downloading and storing computer files are deeply problematic, in my opinion. Whatever the content of those files. (Publication of the contents of the files is another matter. I appreciate how this can be cause for prosecution.)

51 points by popcalc 2y ago

Makes sense and I can see this becoming a global trend if it's not already. But the irony of the news agency taking and posting their photo/video is a facepalm. EDIT: I can foresee a dark future where surveillance video is monetized for such salacious purpose and it becomes inescapable as the cameras are everywhere.

danparsonson 2y ago
That is the dark present in South Korea as they have a huge problem with cameras hidden in hotel rooms and even toilets :-/
Simulacra 2y ago
Could be worse than that. With all of the facial recognition and videoing that's going on by the government, if the police wanted to, they could insert you into any footage committing any crimes they want.

ssalka 2y ago

I could very well be wrong about this, and usually I fall more on the e/acc side, but I think deepfakes are perhaps the gravest immediate concern of AI. Yes, they've been around for well over a decade, but never before has their creation (& thus volume) been so accessible to the average layperson. A simple 10 minutes on CivitAI will tell you all you need to know about how prolific this problem is. IMO it is only a matter of time before this same issue comes to the forefront in the US (I'm kinda surprised it hasn't already become a bigger topic). I could easily see custom celeb LORAs/emmbeddings and realistic SD/SDXL checkpoints being banned outright for this reason. Yes, it would be an infringement on free speech, but weighed against the interests & safety of the public, the courts might see a move like that as reasonable.

transformi 2y ago
Banning Loras/Embedding won't help that..cause you can generate realistic images with only one reference photo these days...
- ssalka 2y ago
  Fair point, technologists will always innovate faster than politicians are able to regulate. Cat's outta the bag now, but that only means the problem will continue to escalate until more drastic measures are deemed necessary. It's that or we will come to accept some level of societal degradation due to the abuse/misuse of AI tools becoming normalized.
deleted 2y ago
[deleted]
slibhb 2y ago
I don't think of myself as on the e/acc side, but I think that deepfakes are here to stay and we'll just have to get used to them. I don't see any way to put the genie back in the bottle but I also don't think it'll take that long to adapt.
- whiplash451 2y ago
  This is giving up a little quickly on watermarking image generation. I do think it’s a hard problem (especially if we want robustness to compression etc) but amenable to research.
m3kw9 2y ago
It won’t be a problem if it gets saturated, people bored of things easily
- ssalka 2y ago
  As AI image generation (and soon video generation) gets to be a more saturated space, we will end up in the state that you cannot trust anything you see online, even if you are educated, well-informed, and meticulously research the topic. It has already been inching that way for a long time, but generative AI definitely is adding new fuel to the fire.

Mistletoe 2y ago

[flagged]

PlunderBunny 2y ago
I think the point is often to harm/exert control over the person depicted, e.g. when boys are making deepfake nudes of girls at their school, as happened in Spain and the USA recently.
skocznymroczny 2y ago
Right now you can generate hundreds of realistic fake nudes of celebrity XYZ and no one cares. Celebrity XYZ accidentally leaks a nude photo from their phone and the entire world loses their mind. I think after the initial moral panic and once the novelty wears off, this issue will fade into obscurity. And currently existing laws should cover sharing such imagery/videos anyway.
bialpio 2y ago
What are your thoughts on hentai then? It's all made up after all. Meta-point is that different people get off on different things, so using seemingly objective language ("the whole point of A is B") to describe something extremely subjective is imo counterproductive.
- Mistletoe 2y ago
  [dead]
anigbrowl 2y ago
Some people don't care about the fakery and are happy with a low quality way to beat off. Other people don't care because their goal is not to get aroused but to embarrass the persons whose real faces have been faked onto the AI-generated or AI-edited porno footage. Some people say 'logically we all know it is fake so just ignore it'. This demonstrates something of a lack of imagination, an inability to picture oneself or a beloved family member being put in that position. It also overlooks that different societies have different social standards and that social mobs tend toward the lowest common IQ denominator. South Korea is a very conservative society, whether you agree with this conservatism or not. Being in a deepfake is easily enough to ruin a person's social or professional reputation, and schoolchildren are especially vulnerable to such pressures. It's not so different from the US problem with sextortion, where teens are tricked into sending nudes to an apparent romantic prospect, and then blackmailed. In both countries, such abuse can end in suicide. In even more conservative countries like Islamic societies, the victims of such fakes could conceivably suffer criminal penalties if authorities don't believe (or deliberately subvert) their protestations of innocence.

blackeyeblitzar 2y ago

[flagged]

thih9 2y ago
> As long as it is not passed off as real, which might be defamatory That’s basically it. AI is extremely accessible and still relatively unknown. Virtually anyone can create a deepfake photo for free in seconds, post it anywhere and a lot of people will assume it is genuine.

SmitheryB 2y ago

I feel as though I'm the only one who doesn't care about this. Curious to know if 20-30 years ago, as graphic design tools got better, were people freaking out about being able to superimpose another person's face on a the body of p*rnstar? Superimposing people's faces on to porn star's bodies happened/s, but does anyone care really even notice or care about it now days? For the most part, of course not. So why should we care about porn videos? What makes them so different? The whole concern smells like typical media hype (i.e. yay - we get to write about AI + Porn - instant clickbait) that nobody will be able to stop and nobody will even care about in 1-2 years: e.g. "oh, yah, another deepfake of <(politician|actress|colleague^)> in a porn film - how incredibly creative". ^ Incidentally, this was made light of in the British version of The Office where a modified image of the boss, David Brent, was sent around the office. If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.

colonelspace 2y ago
It's more problematic than photoshopping a person's face was decades ago because we didn't have pervasive online social media decades ago. Computers weren't in your pocket, and an offending image couldn't be beamed to everyone you know instantly. > If it's joked about harmlessly on mainstream TV, it's probably not a huge concern. There are plenty of mainstream TV jokes about horrendous things. Our ability to joke about something is a poor way to gauge how concerned we should be.
- SmitheryB 2y ago
  > an offending image couldn't be beamed to everyone you know instantly Is your concern that recipients would think it's real? Or is it that people will get an email/chat message containing NSFW material? (because the latter happens now, and is nothing more than background noise at most). What is your actual, specific concern? (genuinely, I have no idea, but want to try to understand - to me it feels like panic about a problem that isn't actually a real problem).
  - colonelspace 2y ago

BadHumans 2y ago

People on HN have a hard time understanding that the barrier to entry matters. Photoshop has existed for decades but the level of skill needed to make realistic looking nudes in Photoshop is not something your average freak with a fetish possesses. New technology puts that power into the hands of every single person.

adventured 2y ago
It's not your average freak with a fetish that is even the primary issue. The volume will come from average anybody (although mostly men). That's the real problem. Random co-workers in the office that want to deepfake their crush. Your average teenage boy with a decent GPU that wants to deepfake their classmates just to experiment. There have already been several instances of this catching the news in Europe for example, so it's obviously going on globally. Oh but the average guy isn't that perverted? Yes they are. We know this from the vast amount of porn they consume and of what type they consume.
- s1artibartfast 2y ago
  Seems like revenge porn and sexual harassment should already cover the use case of distribution. What perverted things someone does with photos in the privacy of their own home seems like it should be their own business to me.
  - ultimafan 2y ago
    Actually nailing someone for revenge porn or sexual harassment seems like a borderline impossible task for police departments if the person doing it has even a minor degree of anonymity. Enlarging the potential pool of people who can do this and the pool of victims they can target by an order of magnitude by lowering both the skill it took to make convincing fakes before and the time taken to make each fake doesn't seem to me like a good idea.
techjamie 2y ago
> New technology puts that power into the hands of every single person. I think the bar is roughly the same, all things considered. If you're going to photoshop, then it's largely a matter of combining two images with a sufficient amount of masking and blurring to make it appear good enough. With image generators, you've got a couple different options. The more trivial would be doing img2img on your target, the more complex being making a LORA. Knowing how tech savvy the average Andy is not, I would say anyone capable of going through the steps to do the AI method could, with some brute-forcing, also manage the Photoshop method. Setting up a local instance of any image generator is beyond the majority of the population, so the closest most can get is paid online GUIs that offer an existing service that they first must become moderately proficient in for good results. Only real difference is that anyone who can do it is much more capable of scale over time than previously. With enough Comfy nodes, you could probably even automate it given face detection and arbitrary images.

BurningFrog 2y ago

When I grew up "a photo never lies" was a rule you could rely on. Yes really!! Photoshop and other advances long since killed that certainty, and now we have a healthy distrust of pictures we're not sure where they came from. I expect videos will now walk a similar path, and life will go on with no major disasters.

singleshot_ 2y ago
When I grew up, my friends dad had a picture taken of his British public school class taken circa 1955. In this photo, which was taken by a slowly rotating panorama camera, a lad started at the beginning of the photo, on the viewers left. Once the camera panned enough that he was out of frame, he sprinted behind the bleachers to the viewers far right, and took his place on the bleachers before the camera panned to take his picture for a second time. Kindly, if you grew up after this picture was taken, and you believed the guideline “a photo never lies,” you were wrong.
Manuel_D 2y ago
Doctored photos have been around for most of the last century. https://en.wikipedia.org/wiki/Censorship_of_images_in_the_Soviet_Union https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
- colonelspace 2y ago
  The appearance of doctored photographs coincides with the appearance of photography.
colonelspace 2y ago
> When I grew up "a photo never lies" was a rule you could rely on. It was a rule you thought you could rely on.
munchler 2y ago
I would refer you to an interesting book called “The Commissar Vanishes: The Falsification of Photographs and Art in Stalin’s Russia”. https://en.wikipedia.org/wiki/The_Commissar_Vanishes https://en.wikipedia.org/wiki/The_Commissar_Vanishes

frognumber 2y ago

I feel like this is a problem without a solution, or at least without a solution more expensive than the problem: * I suspect that, like prohibition or DMCA, criminalizing behavior will lead to worse outcomes than accepting deepfake porn sometimes happens, at least if laws are passed targeting this in isolation. * I suspect that before acceptance, many people will be harmed, on both sides. Humiliated teen girls. Immature hormone-driven teen boys. * I suspect that we'll see a range of bigger problems such as real scams where, for example, a deepfake child calls their elderly parents and empties their bank account. Or, perhaps, a deepfake general calls someone in a nuclear bunker. * Deepfakes are at the tip of technological problems which need solving. We have other things, like the cheap and easy availability of technology to make the next Covid-19. The point is we have the technology to make it in a lab, and the price point is going down by an almost Moore's Law like curve. I don't see a good solution short of completely rethinking democracy and capitalism from scratch. The first time around took the Age of Enlightenment and the Age of Revolution. It's perhaps time to do that. Prove me wrong. Seriously. If there are incremental solutions which stop deepfake porn, deepfake scams, personal drone strikes, and super-viruses, please let me know what they are.

ChrisArchitect 2y ago

Subheading on this describes the story a bit better: South Korea’s national assembly on Thursday passed a Bill to punish people possessing, purchasing, saving or viewing deepfake sexual materials and other fabricated videos.

oska 2y ago
Laws that prosecute ppl for possessing, downloading and storing computer files are deeply problematic, in my opinion. Whatever the content of those files. (Publication of the contents of the files is another matter. I appreciate how this can be cause for prosecution.)