9 ms·

Facebook flooded with 'sextortion' and revenge porn, files reveal

87 points by daenney 9y ago

ShannonAlther 9y ago
> "54,000 potential cases... and 33 of the cases reviewed involved children." That's 0.06% of reviewed cases, which is frankly less than I would have thought. I suspect that child abusers are a more savvy lot than the average chucklehead who does something like this.
- hek3 9y ago
  I am sure there is a ton of stuff going on in the developing world were nobody is used to reporting anything. Edit: Happy to see someone is leaking this stuff given how many flip flops we had to sit threw before someone admitted fake news was a thing.
- LordKano 9y ago
  I'd imagine that the low hanging fruits are picked off by law enforcement pretty quickly.
  - lostlogin 9y ago
    While others drag on for decades with establishment involvement - the U.K. being the current but not first example.
- balls187 9y ago
  "savvy" = not using Facebook.
sergers 9y ago
I am surprised their system for catching bad content is only user reports/flags. While revenge porn and stuff is bad, there are a number of known child related incidents. So if a private group of scumbags, are posting stuff to a private group, noone will know unless someone within reports. Who needs the darkweb when u can have a private Facebook group? Edit: as per replies below, they do use image recognition aswell for I guess known content lists
- DanBC 9y ago
  I suspect they sign up to the IWF lists for filtering.
  - rhizome 9y ago
    International Weightlifting Federation?
    - bdrool 9y ago
      https://www.iwf.org.uk/become-a-member/services-for-members/image-hash-list https://www.iwf.org.uk/become-a-member/services-for-members/... https://www.iwf.org.uk/become-a-member/services-for-members/url-list/url-list-policy https://www.iwf.org.uk/become-a-member/services-for-members/...
- arkitaip 9y ago
  They also use image recognition to detect nudity and previously classified child porn.
  - lightedman 9y ago
    No, they use Magic Number among a bunch of various tricks. You can't really call it image recognition, more like 'data-hash comparison.' To train true image recognition would require a LARGE subset of stored illegal pornography.
    - tomp 9y ago
      Why? Just use two separate datasets, one of images of adults and children, to learn age, the other of porn and non-porn, to learn what's "nudity" or "sex".
      - freeone3000 9y ago
        Knowledge transfer training is really hard. If you can do this (with different categories, for instance, determining what is nudity, then determining black people, to automatically classify porn with black actors automatically), it would be quite impressive. Results have nearly 5% error on CIFAR-10, which is full of clearly distinct categories - planes versus boats.
lightedman 9y ago
Yea, someone tried doing this whole 'sextortion' thing to me on FB. I laughed at them. I used to work in porn, you think you're showing people something they haven't seen (let alone probably want to see again since I 'retired' [AKA got married?]) Sad that shameless people such as myself seem to be immune to this kind of thing. Perhaps everyone should have a career in porn just to eliminate this issue. Give me a break.
frgtpsswrdlame 9y ago
>and the challenges for moderators, who say they are overwhelmed by the volume of work, which means they often have “just 10 seconds” to make a decision. Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines). EDIT: oops, had two articles from the guardian on FB up and referenced the wrong one.
- nickpsecurity 9y ago
  "Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines)." There are systems that can detect porn pretty reliably. Helping moderators spot any porn more quickly will also let them spot revenge porn more quickly. Any correct or false positives can be fed back into detection systems. It might still take more moderators, though. Facebook needs more anyway given the size of user base and money available for moderation.
- bpicolo 9y ago
  > Maybe Facebook should just hire more moderators They are doing exactly that this year
leggomylibro 9y ago
It's disappointing, but not super surprising. Shit like this is why I'd never want to operate a site with a large user base that can freely upload content. Sure you could probably afford the servers for awhile with minimal monetization, but could you pay for even a half-dozen policy experts to moderate the content to a point where you don't send your largest potential audiences running screaming for the hills?
rburhum 9y ago
I had a female friend last month whose ex-bf created a fake account and proceeded to set the profile and public photos and videos of them (having sex). He then went ahead and started sending friend requests to her entire family and friends. I felt so bad for her - it was awful enough to make her even have suicide thoughts. What really surprised me was how horrible the FB system to report stuff like this is. It is basically non-existant. If this thing takes 1 month to be reviewed, that is 1 (eternal) month where the damage can be irreversible. what is worse is that FB has posts about how they are "fighting" this without having any instructions or obvious process defined to report this. So you "report inappropriate content" and the report goes to a black hole while the perpetrator continues with their shenanigans. They dont even need to create a new acct, because they can keep going without the offending account getting blocked. Something else is that the non-profit foundations that exist to fight this (the ones that will send legal take-down notices) are US-centric. This means that if you are from a latin american country and you face this (as was this particular case), you are screwed. Yes obviously the lesson is not to take pics like that in the first place, but sadly a lot of the victims may not be that computer savvy or mature enough to realize what a mistake those things can be.
- rhizome 9y ago
  This isn't about FB solving the problem, it's about FB announcing the existence of their own internal bureaucracies. That is, they won't solve the problem, but they'll create an email address specifically for the problem.
- orblivion 9y ago
  This guy is in the videos as well, right? Could she somehow strike back harder and demolish his reputation? Tell his friends and family, and his employer, what he did to her? Who the hell would send a video of themselves having sex with someone to that person's family?
  - cr0sh 9y ago
    > This guy is in the videos as well, right? I suppose that would depend on how well the guy knows about digital editing of video. ...for that matter, if you wanted to go the opposite way, and you had the right software (and a few other skills) - one could seriously put that guy in a very compromising situation by more digital editing...
gozur88 9y ago
So The Guardian puts this article up just as May's government is promising to tighten government restrictions on social media. That was convenient.
- objclxt 9y ago
  The Guardian isn't a right-wing paper, it's left of center, and I can't remember when it last endorsed the tories. You'd be far more likely to see that kind of thing from The Telegraph or The Daily Mail. There's another explanation much closer to home which doesn't involve Theresa May: The Guardian, like many other newspapers, is under intense pressure from Facebook which is eroding their direct traffic in favor of traffic coming from FB. They literally compete against each other for clicks and ad traffic.
  - gozur88 9y ago
    Sure, but the timing is suspicious. The editors at The Guardian aren't above giving a Tory proposal a little push if it's in their interest. Besides, Labour is proposing essentially the same thing.
nsaslideface 9y ago
Some problems run deeper than their individual manifestations such as these for which one may propose techie solutions -- here, sex ed is the only permanent solution. It's time to find ways to explain to boys that sex is an ordinary way to have fun and/or connect romantically between adult people, rather than a source or arena of power or status as it is endlessly portrayed in media.
- eternalvision 9y ago
  Sex and power/status are inextricably linked for much of society.
  - eternalban 9y ago
    Society implies culture thus construct whereas this feature is observed in much (all?) of animal kingdom as well.
  - wcummings 9y ago
    And that's the problem.
- cat199 9y ago
  To boys? really? Have you ever read the cover of Cosmo or seen clothing with blatant cleavage? The power/control dynamic extends in all directions & I have no doubt that some of this revenge porn extends in the other direction..
bossx 9y ago
They should have a verification process for accounts beyond verifying an email address. Verify identity, it's way too easy for people to be impersonated, there's a whole TV show dedicated to this...
- planteen 9y ago
  You need a phone number to register a new Facebook account...
  - balls187 9y ago
    They're free with Google Voice
    - planteen 9y ago
      I don't think Google Voice will work for that nor will a Trello number. I saw someone complaining on HN that their landline didn't even work.
      - cr0sh 9y ago
        Burner phone?
vivekd 9y ago
So here's my thing: Imagine a world where facebook didn't exist. Would it have been impossible for this guy to get their photos and videos, post them on the internet? I guess facebook does make it easier to email things to family and friends - but it only does so with the user's consent - you have the option to make your friends list private. It seems to me like this guy is a real creep, and he should certainly be punished to the full extent of the law. But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation. The law should certainly step in to prevent stuff like this. But I question how much private companies should have to step in to protect people from their own bad decisions. Yes facebook should certainly take effort to protect it's community standards and prevent porn and revenge porn as a part of it's duty to the community - but does facebook also have an affirmative duty to protect people from outcomes that are a result of their own poor life decisions? Does it owe an affirmative duty to your friend specifically to get rid of all revenge photos about her no matter whom she and how shares her nude or sexual photos? Or is there a duty on your friend to protect herself and exercise discretion in who she shares such content with? Isn't there a duty on your friend to exercise discretion as to whom she sleeps with and whom she allows to take discreet photos and videos with. When she took these photos and videos, she knew there was a possibility that they may be used negatively, so does it negate her responsibility and become completely facebooks problem when those photos are abused by those who misuse their platform? I mean imagine a case where this guy found her bosses and family's phone number and texted the image to her family? Does that then become AT&T's problem to go in there and clean up those images and moderate them?
- barsonme 9y ago
  > But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation. Yeah, this is just classic victim blaming nonsense. Sure, it's good advice to avoid recording anything you don't want eventually published—this goes for conversations as much as it does 'personal' pictures and videos. But that's entirely irrelevant here and sharing 'personal' pictures with your significant other is normal.
brighteyes 9y ago
Facebook disabling 14,000 accounts in a single month because of that kind of terrible behavior does show this is a serious problem. But it's still a very rare problem. Given Facebook has 1.19 billion monthly users, that's on the order of 1 in 100,000 accounts.
- deegles 9y ago
  That seems high to me. We should strive for this to be so rare that it's news-worthy when it happens at all.
  - dec0dedab0de 9y ago
    Well, it's really just an extension of gossip, blame, scandal and shaming that has been around forever. There would need to be a major social change first. Otherwise, even the perfect tech solution would just force this bullshit to move somewhere else.
kyledrake 9y ago
Somebody check my math: Facebook has 1.13 billion dailies, and 1,800 potential daily revenge porn incidents: 1 in 627,777. Involving children, about 1 per day.. 1 in 1.13 billion? Is flooded the right word to describe that? It actually seems pretty low to me.
sctb 9y ago
Previous discussion: https://news.ycombinator.com/item?id=14388102 https://news.ycombinator.com/item?id=14388102

87 points by daenney 9y ago

> "54,000 potential cases... and 33 of the cases reviewed involved children." That's 0.06% of reviewed cases, which is frankly less than I would have thought. I suspect that child abusers are a more savvy lot than the average chucklehead who does something like this.

hek3 9y ago
I am sure there is a ton of stuff going on in the developing world were nobody is used to reporting anything. Edit: Happy to see someone is leaking this stuff given how many flip flops we had to sit threw before someone admitted fake news was a thing.
LordKano 9y ago
I'd imagine that the low hanging fruits are picked off by law enforcement pretty quickly.
- lostlogin 9y ago
  While others drag on for decades with establishment involvement - the U.K. being the current but not first example.
balls187 9y ago
"savvy" = not using Facebook.

sergers 9y ago

I am surprised their system for catching bad content is only user reports/flags. While revenge porn and stuff is bad, there are a number of known child related incidents. So if a private group of scumbags, are posting stuff to a private group, noone will know unless someone within reports. Who needs the darkweb when u can have a private Facebook group? Edit: as per replies below, they do use image recognition aswell for I guess known content lists

DanBC 9y ago
I suspect they sign up to the IWF lists for filtering.
- rhizome 9y ago
  International Weightlifting Federation?
  - bdrool 9y ago
    https://www.iwf.org.uk/become-a-member/services-for-members/image-hash-list https://www.iwf.org.uk/become-a-member/services-for-members/... https://www.iwf.org.uk/become-a-member/services-for-members/url-list/url-list-policy https://www.iwf.org.uk/become-a-member/services-for-members/...
arkitaip 9y ago
They also use image recognition to detect nudity and previously classified child porn.
- lightedman 9y ago
  No, they use Magic Number among a bunch of various tricks. You can't really call it image recognition, more like 'data-hash comparison.' To train true image recognition would require a LARGE subset of stored illegal pornography.
  - tomp 9y ago
    Why? Just use two separate datasets, one of images of adults and children, to learn age, the other of porn and non-porn, to learn what's "nudity" or "sex".
    - freeone3000 9y ago
      Knowledge transfer training is really hard. If you can do this (with different categories, for instance, determining what is nudity, then determining black people, to automatically classify porn with black actors automatically), it would be quite impressive. Results have nearly 5% error on CIFAR-10, which is full of clearly distinct categories - planes versus boats.

lightedman 9y ago

Yea, someone tried doing this whole 'sextortion' thing to me on FB. I laughed at them. I used to work in porn, you think you're showing people something they haven't seen (let alone probably want to see again since I 'retired' [AKA got married?]) Sad that shameless people such as myself seem to be immune to this kind of thing. Perhaps everyone should have a career in porn just to eliminate this issue. Give me a break.

frgtpsswrdlame 9y ago

>and the challenges for moderators, who say they are overwhelmed by the volume of work, which means they often have “just 10 seconds” to make a decision. Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines). EDIT: oops, had two articles from the guardian on FB up and referenced the wrong one.

nickpsecurity 9y ago
"Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines)." There are systems that can detect porn pretty reliably. Helping moderators spot any porn more quickly will also let them spot revenge porn more quickly. Any correct or false positives can be fed back into detection systems. It might still take more moderators, though. Facebook needs more anyway given the size of user base and money available for moderation.
bpicolo 9y ago
> Maybe Facebook should just hire more moderators They are doing exactly that this year

leggomylibro 9y ago

It's disappointing, but not super surprising. Shit like this is why I'd never want to operate a site with a large user base that can freely upload content. Sure you could probably afford the servers for awhile with minimal monetization, but could you pay for even a half-dozen policy experts to moderate the content to a point where you don't send your largest potential audiences running screaming for the hills?

rburhum 9y ago

I had a female friend last month whose ex-bf created a fake account and proceeded to set the profile and public photos and videos of them (having sex). He then went ahead and started sending friend requests to her entire family and friends. I felt so bad for her - it was awful enough to make her even have suicide thoughts. What really surprised me was how horrible the FB system to report stuff like this is. It is basically non-existant. If this thing takes 1 month to be reviewed, that is 1 (eternal) month where the damage can be irreversible. what is worse is that FB has posts about how they are "fighting" this without having any instructions or obvious process defined to report this. So you "report inappropriate content" and the report goes to a black hole while the perpetrator continues with their shenanigans. They dont even need to create a new acct, because they can keep going without the offending account getting blocked. Something else is that the non-profit foundations that exist to fight this (the ones that will send legal take-down notices) are US-centric. This means that if you are from a latin american country and you face this (as was this particular case), you are screwed. Yes obviously the lesson is not to take pics like that in the first place, but sadly a lot of the victims may not be that computer savvy or mature enough to realize what a mistake those things can be.

rhizome 9y ago
This isn't about FB solving the problem, it's about FB announcing the existence of their own internal bureaucracies. That is, they won't solve the problem, but they'll create an email address specifically for the problem.
orblivion 9y ago
This guy is in the videos as well, right? Could she somehow strike back harder and demolish his reputation? Tell his friends and family, and his employer, what he did to her? Who the hell would send a video of themselves having sex with someone to that person's family?
- cr0sh 9y ago
  > This guy is in the videos as well, right? I suppose that would depend on how well the guy knows about digital editing of video. ...for that matter, if you wanted to go the opposite way, and you had the right software (and a few other skills) - one could seriously put that guy in a very compromising situation by more digital editing...

gozur88 9y ago

So The Guardian puts this article up just as May's government is promising to tighten government restrictions on social media. That was convenient.

objclxt 9y ago
The Guardian isn't a right-wing paper, it's left of center, and I can't remember when it last endorsed the tories. You'd be far more likely to see that kind of thing from The Telegraph or The Daily Mail. There's another explanation much closer to home which doesn't involve Theresa May: The Guardian, like many other newspapers, is under intense pressure from Facebook which is eroding their direct traffic in favor of traffic coming from FB. They literally compete against each other for clicks and ad traffic.
- gozur88 9y ago
  Sure, but the timing is suspicious. The editors at The Guardian aren't above giving a Tory proposal a little push if it's in their interest. Besides, Labour is proposing essentially the same thing.

nsaslideface 9y ago

Some problems run deeper than their individual manifestations such as these for which one may propose techie solutions -- here, sex ed is the only permanent solution. It's time to find ways to explain to boys that sex is an ordinary way to have fun and/or connect romantically between adult people, rather than a source or arena of power or status as it is endlessly portrayed in media.

eternalvision 9y ago
Sex and power/status are inextricably linked for much of society.
- eternalban 9y ago
  Society implies culture thus construct whereas this feature is observed in much (all?) of animal kingdom as well.
- wcummings 9y ago
  And that's the problem.
cat199 9y ago
To boys? really? Have you ever read the cover of Cosmo or seen clothing with blatant cleavage? The power/control dynamic extends in all directions & I have no doubt that some of this revenge porn extends in the other direction..

bossx 9y ago

They should have a verification process for accounts beyond verifying an email address. Verify identity, it's way too easy for people to be impersonated, there's a whole TV show dedicated to this...

planteen 9y ago
You need a phone number to register a new Facebook account...
- balls187 9y ago
  They're free with Google Voice
  - planteen 9y ago
    I don't think Google Voice will work for that nor will a Trello number. I saw someone complaining on HN that their landline didn't even work.
    - cr0sh 9y ago
      Burner phone?

vivekd 9y ago

So here's my thing: Imagine a world where facebook didn't exist. Would it have been impossible for this guy to get their photos and videos, post them on the internet? I guess facebook does make it easier to email things to family and friends - but it only does so with the user's consent - you have the option to make your friends list private. It seems to me like this guy is a real creep, and he should certainly be punished to the full extent of the law. But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation. The law should certainly step in to prevent stuff like this. But I question how much private companies should have to step in to protect people from their own bad decisions. Yes facebook should certainly take effort to protect it's community standards and prevent porn and revenge porn as a part of it's duty to the community - but does facebook also have an affirmative duty to protect people from outcomes that are a result of their own poor life decisions? Does it owe an affirmative duty to your friend specifically to get rid of all revenge photos about her no matter whom she and how shares her nude or sexual photos? Or is there a duty on your friend to protect herself and exercise discretion in who she shares such content with? Isn't there a duty on your friend to exercise discretion as to whom she sleeps with and whom she allows to take discreet photos and videos with. When she took these photos and videos, she knew there was a possibility that they may be used negatively, so does it negate her responsibility and become completely facebooks problem when those photos are abused by those who misuse their platform? I mean imagine a case where this guy found her bosses and family's phone number and texted the image to her family? Does that then become AT&T's problem to go in there and clean up those images and moderate them?

barsonme 9y ago
> But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation. Yeah, this is just classic victim blaming nonsense. Sure, it's good advice to avoid recording anything you don't want eventually published—this goes for conversations as much as it does 'personal' pictures and videos. But that's entirely irrelevant here and sharing 'personal' pictures with your significant other is normal.

brighteyes 9y ago

Facebook disabling 14,000 accounts in a single month because of that kind of terrible behavior does show this is a serious problem. But it's still a very rare problem. Given Facebook has 1.19 billion monthly users, that's on the order of 1 in 100,000 accounts.

deegles 9y ago
That seems high to me. We should strive for this to be so rare that it's news-worthy when it happens at all.
- dec0dedab0de 9y ago
  Well, it's really just an extension of gossip, blame, scandal and shaming that has been around forever. There would need to be a major social change first. Otherwise, even the perfect tech solution would just force this bullshit to move somewhere else.

kyledrake 9y ago

Somebody check my math: Facebook has 1.13 billion dailies, and 1,800 potential daily revenge porn incidents: 1 in 627,777. Involving children, about 1 per day.. 1 in 1.13 billion? Is flooded the right word to describe that? It actually seems pretty low to me.

sctb 9y ago

Previous discussion: https://news.ycombinator.com/item?id=14388102 https://news.ycombinator.com/item?id=14388102