11 ms·

Strengthening our efforts against the spread of non-consensual intimate images

80 points by arkadiyt 5y ago

cstejerean 5y ago
Could this be used by someone to stop sharing any image or video of themselves they don’t like? Or does the tech enforce in some way that only NCII content is eligible? If so I’m curious how that works.
- bellyfullofbac 5y ago
  I guess it'll be a "Facebook censorship API". Beyonce wants an image to disappear? If it's not on FB, IG, WhatsApp, Pinterest, it's pretty good. Prince Andrew doesn't want a pic of him with his hands around a minor and Ghislaine Maxwell nearby? Call Zuck...
- charcircuit 5y ago
  It only sends hashes of the contents so I don't know how they would tell unless they review what images are being flagged by these hashes after it gets included.
- dcallies 5y ago
  Another commenter (“ahahahahah”) described it as the equivalent of a “report image” button, which I thought was apt - it preemptively reports on all the platforms that are participating with StopNCII. It comes down to each platform: Meta has specific rules against NCII (https://transparency.fb.com/policies/community-standards/sexual-exploitation-adults/ https://transparency.fb.com/policies/community-standards/sex...) - we prioritize potential NCII content as high severity and requiring immediate review to verify. However, just like hitting the report button on each of those platforms, they might have different rules.
dplarson 5y ago
If it uses a hash of an image, does that mean an edited image (eg cropped or resized) wouldn't be detected? Or is there a way to extend a single hash to multiple image variants? I imagine the answer is no, but I would also like to be optimistic and think even detecting the original version would stop a lot of NCII sharing (ie it's not a perfect solution, but still helpful).
- charcircuit 5y ago
  It uses PDQ for images and MD5 for videos. I imagine it's trivial to bypass.
  - dcallies 5y ago
    Yup, PDQ (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5- https://stopncii.org/faq/ https://stopncii.org/faq/. MD5 is a cryptographic hash, which means that even a single bit changed gives you an entirely different hash. This usually precludes any attacks to try and deliberately generate collisions, but also means it's harder to match “benign” changes - for example, many platforms re-encode videos as part of the upload process (reducing resolution, changing formats, etc). However, many platforms skip re-encoding, so MD5 can have better results than you might think at first glance. The more bits you take from the original content, the harder you can make it to bypass, but the more you might have to worry about capturing enough that you might compromise the privacy of the original submitter. For the program, StopNCII picked a set of tradeoffs between those two tensions, and are keeping a close eye on how effective it is, and will iterate if need be.
- notacoward 5y ago
  > does that mean an edited image (eg cropped or resized) wouldn't be detected? Depends on the hash, and I also wouldn't be surprised if "fingerprint" turned out to be a more accurate term. This is a PR document, after all. ;) You can normalize things like resolution and color balance, and create a hash/fingerprint that's a concatenation of hashes for portions of the image, which would allow some degree of useful comparison despite many kinds of changes. Similar techniques have been used to detect copyright violations as well, so there are technological precedents. The real question is implementation quality.
- lkschubert8 5y ago
  Here's an article on how a few different image hashing algorithms work https://content-blockchain.org/research/testing-different-image-hash-functions/ https://content-blockchain.org/research/testing-different-im.... I think in general they still function with resizing (and a few other image manipulations), but I don't think they handle cropping.
ludamad 5y ago
There's a gray area where someone consented regrettably, say as a brief stint on OnlyFans until they realized how pictures travel. I wonder how Meta will rule on those
- polyomino 5y ago
  That's not a gray area. Consent can be withdrawn at any time.
  - salt-thrower 5y ago
    Consent during a sex act and consent to an image that you created continuing to exist after you knowingly release it into the wild are two different things. (Not including revenge porn or other nonconsensual recordings, obviously). An actor who performs in an embarrassingly bad movie might wish they could refuse to let their work be used after the fact, but they already signed away their right to retract their work. An OnlyFans model is also the creator of the content though, so they might have more leverage depending on OF's legal terms. If they still own the rights to all media they create and distribute through OF, then they could petition for removal based on DMCA or something (I'm not a lawyer but it sounds plausible). Morally, I hope FB can help regretful adult models remove content of themselves if they don't want it to exist anymore. But practically, that will be very difficult to guarantee.
  - megous 5y ago
    Consent? Yes. Images? No.
  - A4ET8a8uTh0 5y ago
    Any time? Can you withdraw consent after the fact and claim rape? If you are going to use categorical language, please be more careful about the language you use to describe a black and white world.
  - khazhoux 5y ago
    The platform can stop hosting some content immediately once consent is withdrawn, but it can't prevent that content from having already escaped. It's probably not technically a "gray area" since the platform can't ensure the content doesn't escape. But certainly, content creators should be aware that withdrawing consent doesn't mean their content is suddenly secured.
slg 5y ago
So what stops someone from uploading an image that isn't NCII as an attempt to censor an image or get whoever shared the image in trouble? I imagine there needs to be human oversight in some way to stop abuse. But does that mean that in order to stop the spread of your personal NCII, you need to proactively share those exact same photos you are worried about getting shared?
- hypertele-Xii 5y ago
  Upload your photo here to check if it's on the Internet, for free!
  - ciphol 5y ago
    You could upload part of the picture (with the nudity, or everything personally identifying, cropped out) and see if it matches part of any other picture online.
  - a_imho 5y ago
    This frequently happens with GDPR data/delete requests. They will ask you to verify all the shadow data they collected about you is indeed correct before giving you a summary or promise to delete it.
    - OJFord 5y ago
      PayPal won't delete an account I didn't knowingly consent (I'm sure there was some hidden terms hiding behind some dark UX pattern) to being created unless I prove my identity with information that isn't already recorded on the account. At least, not visible to me. All that's there is a name and bank account link, I told them 'I am Oliver Ford' and that I'm happy to provide a secret shared by way of reference attached to a penny transfer to that linked account; but no - I must provide a copy of my passport and proof of address (which they don't know, and I can edit on the site anyway), obviously.
      - a_imho 5y ago
quacksilver 5y ago
As the images never leave the device (and you can't build an image from the hash) then surely they are dependent on the victim certifying that the image is a picture of them. They have just made a tool to ban 20 arbitrary images at a time from social media that is gated by some questions that anyone can lie to. I wonder if you could interact with the API directly if you were feeling evil Real revenge porn uploaders could just change a pixel / reencode / tint the image like you do for flesh tone detection
- IAmEveryone 5y ago
  A perceptual hash is immune against the sort of manipulation you mention. One method I’ve implemented myself is recording changes in brightness along a path in the image. Inverting colors would work, but also make the image somewhat worthless. Flipping would also work, but is usually protected against by adding the flipped hash as well.
  - quacksilver 5y ago
    True. I am not sure exactly what hash is being used (though others seem to indicate md5) I have heard from colleagues in the past that consumers of illegal pornography used to (or possibly still do) commonly apply a negative filter or hue shift to images to avoid hash matches and flesh tone detection. This helps when combined with other stuff if someone scans their disk or they get raided and have too much stuff for law enforcement to look through properly. They then set up their screen or display settings to counteract the filter when viewing it. Thankfully I have no experience with illegal pornography and definitely don't want to gain any, so hopefully I will never be able to confirm that firsthand.
    - dcallies 5y ago
      It’s somewhere in the FAQ at https://stopncii.org/faq/ https://stopncii.org/faq/, but It’s PDQ for photos (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5 for videos. PDQ is resistant to some modifications (it focuses on the ones that come from regular usage, such as changing the format from gif to jpg, or a filter changing colors or brightness), but it’s not as resistant to modifications as you could get by training dedicated classifiers or other approaches that you might do with the original media or by storing more context, which StopNCII chose not to do.
Hizonner 5y ago
For those of you who are confused, this is about revenge porn. Sometimes unfortunately referred to as "Nonconsensual Intimate Images". While we're at it, it's "child porn", not "CSAM". These weird jargon names aren't intrinsically less offensive. They don't protect anybody from anything. They certainly don't address any actual problem. They're not necessarily more technically correct, and in fact the word "intimate" in this particular one brings in a ton of baggage that basically guarantees it will be technically wrong much more often than the original, intuitive phrase. They're just more confusing. They basically exist so that people can feel superior for using them. People who do use them are acting badly and should feel bad.
- zffr 5y ago
  There are quite a few organizations that take the opposite stance: * https://www.inhope.org/EN/articles/child-sexual-abuse-material?locale=en https://www.inhope.org/EN/articles/child-sexual-abuse-materi... * https://childrescuecoalition.org/educations/call-it-what-it-is-help-us-change-legislation-from-child-pornography-to-csam/ https://childrescuecoalition.org/educations/call-it-what-it-... * https://www.thorn.org/blog/decoding-the-language-of-child-sexual-exploitation-acronyms-to-know/ https://www.thorn.org/blog/decoding-the-language-of-child-se... * https://www.adfsolutions.com/news/what-is-csam https://www.adfsolutions.com/news/what-is-csam
  - Natsu 5y ago
    I get what they're gesturing at, but I find a stronger emotional reaction against child porn than CSAM and it never once occurred to me that anyone thought this term implied it was in any way consensual or non-horrific.
    - acdha 5y ago
      It’s certainly a more familiar term for many of us, but I prefer the new terms because “porn” has pleasurable connotations for most people. Using that term focuses on the abusers’ pleasure rather than the victims’ suffering. I think this is especially important at older ages: if you remember things like https://en.wikipedia.org/wiki/2014_celebrity_nude_photo_leak https://en.wikipedia.org/wiki/2014_celebrity_nude_photo_leak there were entirely too many men who felt comfortable making public statements about how great it was (I remember seeing some open source developers tweeting about this like it was a gift to the world). That won’t happen for young children but it definitely does for teenagers, so I’m fine with changing the language to focus on the horrific experience.
kawsper 5y ago
I hope the technology can help in cases like the one described here: https://www.thetimes.co.uk/article/denmark-gives-facebook-24h-limit-for-deleting-illegal-posts-cs7pc0ldp https://www.thetimes.co.uk/article/denmark-gives-facebook-24... A danish tourist was killed on holiday and it was filmed (very graphic) and now the parents of the deceased are being sent videos. > She said she had reported every instance of the harassment to Facebook but in a number of cases it had left the video online. She also complained that Danish police had failed to identify the culprit. > Simon Kollerup, 35, the Danish trade minister responsible for internet regulation, said Facebook’s “hopeless” response had underscored the need for more stringent EU rules on social media. > Martin Ruby, 48, Facebook’s head of public policy for the Nordic and Benelux regions, said he was sorry “if we made the wrong calls in this case” but insisted that it had done its best to eradicate the execution clips. He said that the code for the videos was being continually tweaked to elude Facebook’s censorship algorithms. “The problem is that these evil people are sitting there and changing the format on them,” he told DR. “It’s a perpetual battle.”
- animal_spirits 5y ago
  Absolutely terrible
- aaron695 5y ago
- klabb3 5y ago
  What, corporate community guidelines aren't serving the purpose they're supposedly for? I'm shocked!
- echelon 5y ago
  I'm usually a free speech absolutist and privacy advocate, but whomever is sending that poor woman's parents videos of her torture and beheading deserves to be banned from the Internet forever. At minimum. I can't imagine how those poor people must feel.
  - causi 5y ago
literallyaduck 5y ago
Would it not be easy to stop the spread of intimate images through your platform? If you have a platform for "intimate" images maybe have a record on file that shows everyone is of age and consents. If an image, or video providence can not be confirmed you don't put it up for consumption. There is no shortage of "intimate" images and it would not present a hardship or be a barrier to entry to those that want to produce or consume the content. Please, explain how it is ethical to do otherwise.
csmpltn 5y ago
> "Once you have created your hash and submitted it to the StopNCII.org bank it will be sent out to participating companies’ platforms. If someone tries to upload a matching image, the platforms will review the content to check if it violates their policies and take action accordingly." So what stops somebody from uploading a benign image? Say, an image of the Statue of Liberty? Will the platform propagate the hashes? Will it proceed to flag similar images found on the devices of people around the world?
- ahahahahah 5y ago
  Nothing. But also nothing prevents a user from manually reporting an image of the Statue of Liberty as a violation of such a policy. At best, this could be used to help the user automate such reports. Such abuse (that of users reporting benign images) is far from the most difficult abuse problems facing these platforms, though it does have the effect of making fighting "real" abuse harder.
  - dcallies 5y ago
    I like the comparison of the program to hitting “report image” on a platform. Advantages of StopNCII is that you can report proactively, and to report it to all the participating platforms at the same time. Misuse of the StopNCII platform is something that we (Meta/Facebook), UK Revenge Porn Helpline and the full StopNCII team discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-positives, because now you can use your platform-specific ML, or in-house photo detection algorithms.
    - csmpltn 5y ago
      > "Misuse of the StopNCII platform is something that we (Meta/Facebook) discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-po" Okay, and what was the conclusion then? Can I use the service to take down arbitrary content online that has nothing to do with the subject at hand? Can I just incriminate other devices, users and accounts that store innocent data and get it deleted without anybody's consent?
nullc 5y ago
I clicked on some random reddit user a few days ago and his top comments were on some thread with a bunch of nude images from some person whos accounts were hacked, his comments were gloating about the thread being a top google hit on her name and that people at her university and prospective employers would likely find the images. I reported the posts and the thread and reddit sent back a form letter response: "Thanks for submitting a report to the Reddit admin team. After investigating, we’ve found that the reported content doesn’t violate Reddit’s Content Policy." The absentee management of these platforms has turned them into an attractive nuisance. Getting more reports won't address the underlying issues.
OneLeggedCat 5y ago
Non-consensual tracking, on the other hand...
verisimi 5y ago
Come on, let's not beat around the bush. The answer to this is that we need to log on using our biomedical-crypto-citizen-id, just to go online. Everything we do _needs_ to be fully tracked. We need to serve our governments all the information about everyone, so that they can catch the baddies* - its essential! It really is true - someone did something bad online, so that justifies the loss of liberty for everyone! Get with it. Roll over. Right?!? * exceptions apply for key governmental employees.
- croon 5y ago
  You kid, but the dystopia of 1984 is probably pretty close to the utopia where everyones rights are protected while also being held accountable by a benevolent technocracy. I just understand that the latter won't last a day before it's abused and devolves into the former.
- ydlr 5y ago
  > * exceptions apply for key governmental employees. If government employees are exempt, where would they get the evidence needed to blackmail each other into maintaining the system.
tzs 5y ago
I sometimes think that we might need to approach this from the opposite direction. Declare a national "post your privates" day where we all post to all our social media pictures pictures of ourselves nude and/or engaging in sexual activity. Make this a regular event. Try to make it something everyone participates in. Fat people. Skinny people. People who climb on rocks. Tough people. Sissy people. Even people with chicken pox. Elderly people. Kids (but just nudes, nothing sexual). Singles. Couples. Whole families. Outside of the "post your junk" days social media would still ban such material. The idea is to make it so that if someone then does post intimate pictures of you without your permission the impact would be about as bad as someone posting an unauthorized picture of you having lunch. That wouldn't completely defang such photos of course, because sometimes the harm from the photo is not that it shows your private parts or shows you having sex but rather who it shows you with when you were doing it. A photo of Juliet with Romeo would have caused her trouble even if they were fully clothed and just talking. But for the cases where what makes the images damaging is the nudity or sex itself, normalizing this could help.
- kelseyfrog 5y ago
  This is like posting your SSN to make yourself immune to identity theft.
huetius 5y ago
Honest, if difficult, questions: Is there any benefit to legalized pornography that outweighs the horror of things like this? Does the amount of bureaucratic scaffolding and invasive surveillance by corporate and government actors required to police the boundary of consensual and non-consensual pornography leave you substantially more free than just punishing platforms which permit the dissemination of pornographic images?
dcallies 5y ago
Hey everyone! I’m a software engineer at Meta, and I helped work on the program that the newsroom post is about. Sorry I’m late, I only just found the thread! I might be able to answer some questions about the technology - many of the building blocks are open source, like the PDQ photo hashing algorithm (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) mentioned in FAQ (https://stopncii.org/faq/ https://stopncii.org/faq/). I might not be able to answer every question, but I’ll do my best!

80 points by arkadiyt 5y ago

Could this be used by someone to stop sharing any image or video of themselves they don’t like? Or does the tech enforce in some way that only NCII content is eligible? If so I’m curious how that works.

bellyfullofbac 5y ago
I guess it'll be a "Facebook censorship API". Beyonce wants an image to disappear? If it's not on FB, IG, WhatsApp, Pinterest, it's pretty good. Prince Andrew doesn't want a pic of him with his hands around a minor and Ghislaine Maxwell nearby? Call Zuck...
charcircuit 5y ago
It only sends hashes of the contents so I don't know how they would tell unless they review what images are being flagged by these hashes after it gets included.
dcallies 5y ago
Another commenter (“ahahahahah”) described it as the equivalent of a “report image” button, which I thought was apt - it preemptively reports on all the platforms that are participating with StopNCII. It comes down to each platform: Meta has specific rules against NCII (https://transparency.fb.com/policies/community-standards/sexual-exploitation-adults/ https://transparency.fb.com/policies/community-standards/sex...) - we prioritize potential NCII content as high severity and requiring immediate review to verify. However, just like hitting the report button on each of those platforms, they might have different rules.

dplarson 5y ago

If it uses a hash of an image, does that mean an edited image (eg cropped or resized) wouldn't be detected? Or is there a way to extend a single hash to multiple image variants? I imagine the answer is no, but I would also like to be optimistic and think even detecting the original version would stop a lot of NCII sharing (ie it's not a perfect solution, but still helpful).

charcircuit 5y ago
It uses PDQ for images and MD5 for videos. I imagine it's trivial to bypass.
- dcallies 5y ago
  Yup, PDQ (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5- https://stopncii.org/faq/ https://stopncii.org/faq/. MD5 is a cryptographic hash, which means that even a single bit changed gives you an entirely different hash. This usually precludes any attacks to try and deliberately generate collisions, but also means it's harder to match “benign” changes - for example, many platforms re-encode videos as part of the upload process (reducing resolution, changing formats, etc). However, many platforms skip re-encoding, so MD5 can have better results than you might think at first glance. The more bits you take from the original content, the harder you can make it to bypass, but the more you might have to worry about capturing enough that you might compromise the privacy of the original submitter. For the program, StopNCII picked a set of tradeoffs between those two tensions, and are keeping a close eye on how effective it is, and will iterate if need be.
notacoward 5y ago
> does that mean an edited image (eg cropped or resized) wouldn't be detected? Depends on the hash, and I also wouldn't be surprised if "fingerprint" turned out to be a more accurate term. This is a PR document, after all. ;) You can normalize things like resolution and color balance, and create a hash/fingerprint that's a concatenation of hashes for portions of the image, which would allow some degree of useful comparison despite many kinds of changes. Similar techniques have been used to detect copyright violations as well, so there are technological precedents. The real question is implementation quality.
lkschubert8 5y ago
Here's an article on how a few different image hashing algorithms work https://content-blockchain.org/research/testing-different-image-hash-functions/ https://content-blockchain.org/research/testing-different-im.... I think in general they still function with resizing (and a few other image manipulations), but I don't think they handle cropping.

ludamad 5y ago

There's a gray area where someone consented regrettably, say as a brief stint on OnlyFans until they realized how pictures travel. I wonder how Meta will rule on those

polyomino 5y ago
That's not a gray area. Consent can be withdrawn at any time.
- salt-thrower 5y ago
  Consent during a sex act and consent to an image that you created continuing to exist after you knowingly release it into the wild are two different things. (Not including revenge porn or other nonconsensual recordings, obviously). An actor who performs in an embarrassingly bad movie might wish they could refuse to let their work be used after the fact, but they already signed away their right to retract their work. An OnlyFans model is also the creator of the content though, so they might have more leverage depending on OF's legal terms. If they still own the rights to all media they create and distribute through OF, then they could petition for removal based on DMCA or something (I'm not a lawyer but it sounds plausible). Morally, I hope FB can help regretful adult models remove content of themselves if they don't want it to exist anymore. But practically, that will be very difficult to guarantee.
- megous 5y ago
  Consent? Yes. Images? No.
- A4ET8a8uTh0 5y ago
  Any time? Can you withdraw consent after the fact and claim rape? If you are going to use categorical language, please be more careful about the language you use to describe a black and white world.
- khazhoux 5y ago
  The platform can stop hosting some content immediately once consent is withdrawn, but it can't prevent that content from having already escaped. It's probably not technically a "gray area" since the platform can't ensure the content doesn't escape. But certainly, content creators should be aware that withdrawing consent doesn't mean their content is suddenly secured.

slg 5y ago

So what stops someone from uploading an image that isn't NCII as an attempt to censor an image or get whoever shared the image in trouble? I imagine there needs to be human oversight in some way to stop abuse. But does that mean that in order to stop the spread of your personal NCII, you need to proactively share those exact same photos you are worried about getting shared?

hypertele-Xii 5y ago
Upload your photo here to check if it's on the Internet, for free!
- ciphol 5y ago
  You could upload part of the picture (with the nudity, or everything personally identifying, cropped out) and see if it matches part of any other picture online.
- a_imho 5y ago
  This frequently happens with GDPR data/delete requests. They will ask you to verify all the shadow data they collected about you is indeed correct before giving you a summary or promise to delete it.
  - OJFord 5y ago
    PayPal won't delete an account I didn't knowingly consent (I'm sure there was some hidden terms hiding behind some dark UX pattern) to being created unless I prove my identity with information that isn't already recorded on the account. At least, not visible to me. All that's there is a name and bank account link, I told them 'I am Oliver Ford' and that I'm happy to provide a secret shared by way of reference attached to a penny transfer to that linked account; but no - I must provide a copy of my passport and proof of address (which they don't know, and I can edit on the site anyway), obviously.
    - a_imho 5y ago

quacksilver 5y ago

As the images never leave the device (and you can't build an image from the hash) then surely they are dependent on the victim certifying that the image is a picture of them. They have just made a tool to ban 20 arbitrary images at a time from social media that is gated by some questions that anyone can lie to. I wonder if you could interact with the API directly if you were feeling evil Real revenge porn uploaders could just change a pixel / reencode / tint the image like you do for flesh tone detection

IAmEveryone 5y ago
A perceptual hash is immune against the sort of manipulation you mention. One method I’ve implemented myself is recording changes in brightness along a path in the image. Inverting colors would work, but also make the image somewhat worthless. Flipping would also work, but is usually protected against by adding the flipped hash as well.
- quacksilver 5y ago
  True. I am not sure exactly what hash is being used (though others seem to indicate md5) I have heard from colleagues in the past that consumers of illegal pornography used to (or possibly still do) commonly apply a negative filter or hue shift to images to avoid hash matches and flesh tone detection. This helps when combined with other stuff if someone scans their disk or they get raided and have too much stuff for law enforcement to look through properly. They then set up their screen or display settings to counteract the filter when viewing it. Thankfully I have no experience with illegal pornography and definitely don't want to gain any, so hopefully I will never be able to confirm that firsthand.
  - dcallies 5y ago
    It’s somewhere in the FAQ at https://stopncii.org/faq/ https://stopncii.org/faq/, but It’s PDQ for photos (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5 for videos. PDQ is resistant to some modifications (it focuses on the ones that come from regular usage, such as changing the format from gif to jpg, or a filter changing colors or brightness), but it’s not as resistant to modifications as you could get by training dedicated classifiers or other approaches that you might do with the original media or by storing more context, which StopNCII chose not to do.

Hizonner 5y ago

For those of you who are confused, this is about revenge porn. Sometimes unfortunately referred to as "Nonconsensual Intimate Images". While we're at it, it's "child porn", not "CSAM". These weird jargon names aren't intrinsically less offensive. They don't protect anybody from anything. They certainly don't address any actual problem. They're not necessarily more technically correct, and in fact the word "intimate" in this particular one brings in a ton of baggage that basically guarantees it will be technically wrong much more often than the original, intuitive phrase. They're just more confusing. They basically exist so that people can feel superior for using them. People who do use them are acting badly and should feel bad.

zffr 5y ago
There are quite a few organizations that take the opposite stance: * https://www.inhope.org/EN/articles/child-sexual-abuse-material?locale=en https://www.inhope.org/EN/articles/child-sexual-abuse-materi... * https://childrescuecoalition.org/educations/call-it-what-it-is-help-us-change-legislation-from-child-pornography-to-csam/ https://childrescuecoalition.org/educations/call-it-what-it-... * https://www.thorn.org/blog/decoding-the-language-of-child-sexual-exploitation-acronyms-to-know/ https://www.thorn.org/blog/decoding-the-language-of-child-se... * https://www.adfsolutions.com/news/what-is-csam https://www.adfsolutions.com/news/what-is-csam
- Natsu 5y ago
  I get what they're gesturing at, but I find a stronger emotional reaction against child porn than CSAM and it never once occurred to me that anyone thought this term implied it was in any way consensual or non-horrific.
  - acdha 5y ago
    It’s certainly a more familiar term for many of us, but I prefer the new terms because “porn” has pleasurable connotations for most people. Using that term focuses on the abusers’ pleasure rather than the victims’ suffering. I think this is especially important at older ages: if you remember things like https://en.wikipedia.org/wiki/2014_celebrity_nude_photo_leak https://en.wikipedia.org/wiki/2014_celebrity_nude_photo_leak there were entirely too many men who felt comfortable making public statements about how great it was (I remember seeing some open source developers tweeting about this like it was a gift to the world). That won’t happen for young children but it definitely does for teenagers, so I’m fine with changing the language to focus on the horrific experience.

kawsper 5y ago

I hope the technology can help in cases like the one described here: https://www.thetimes.co.uk/article/denmark-gives-facebook-24h-limit-for-deleting-illegal-posts-cs7pc0ldp https://www.thetimes.co.uk/article/denmark-gives-facebook-24... A danish tourist was killed on holiday and it was filmed (very graphic) and now the parents of the deceased are being sent videos. > She said she had reported every instance of the harassment to Facebook but in a number of cases it had left the video online. She also complained that Danish police had failed to identify the culprit. > Simon Kollerup, 35, the Danish trade minister responsible for internet regulation, said Facebook’s “hopeless” response had underscored the need for more stringent EU rules on social media. > Martin Ruby, 48, Facebook’s head of public policy for the Nordic and Benelux regions, said he was sorry “if we made the wrong calls in this case” but insisted that it had done its best to eradicate the execution clips. He said that the code for the videos was being continually tweaked to elude Facebook’s censorship algorithms. “The problem is that these evil people are sitting there and changing the format on them,” he told DR. “It’s a perpetual battle.”

animal_spirits 5y ago
Absolutely terrible
aaron695 5y ago
klabb3 5y ago
What, corporate community guidelines aren't serving the purpose they're supposedly for? I'm shocked!
echelon 5y ago
I'm usually a free speech absolutist and privacy advocate, but whomever is sending that poor woman's parents videos of her torture and beheading deserves to be banned from the Internet forever. At minimum. I can't imagine how those poor people must feel.
- causi 5y ago

literallyaduck 5y ago

Would it not be easy to stop the spread of intimate images through your platform? If you have a platform for "intimate" images maybe have a record on file that shows everyone is of age and consents. If an image, or video providence can not be confirmed you don't put it up for consumption. There is no shortage of "intimate" images and it would not present a hardship or be a barrier to entry to those that want to produce or consume the content. Please, explain how it is ethical to do otherwise.

csmpltn 5y ago

> "Once you have created your hash and submitted it to the StopNCII.org bank it will be sent out to participating companies’ platforms. If someone tries to upload a matching image, the platforms will review the content to check if it violates their policies and take action accordingly." So what stops somebody from uploading a benign image? Say, an image of the Statue of Liberty? Will the platform propagate the hashes? Will it proceed to flag similar images found on the devices of people around the world?

ahahahahah 5y ago
Nothing. But also nothing prevents a user from manually reporting an image of the Statue of Liberty as a violation of such a policy. At best, this could be used to help the user automate such reports. Such abuse (that of users reporting benign images) is far from the most difficult abuse problems facing these platforms, though it does have the effect of making fighting "real" abuse harder.
- dcallies 5y ago
  I like the comparison of the program to hitting “report image” on a platform. Advantages of StopNCII is that you can report proactively, and to report it to all the participating platforms at the same time. Misuse of the StopNCII platform is something that we (Meta/Facebook), UK Revenge Porn Helpline and the full StopNCII team discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-positives, because now you can use your platform-specific ML, or in-house photo detection algorithms.
  - csmpltn 5y ago
    > "Misuse of the StopNCII platform is something that we (Meta/Facebook) discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-po" Okay, and what was the conclusion then? Can I use the service to take down arbitrary content online that has nothing to do with the subject at hand? Can I just incriminate other devices, users and accounts that store innocent data and get it deleted without anybody's consent?

nullc 5y ago

I clicked on some random reddit user a few days ago and his top comments were on some thread with a bunch of nude images from some person whos accounts were hacked, his comments were gloating about the thread being a top google hit on her name and that people at her university and prospective employers would likely find the images. I reported the posts and the thread and reddit sent back a form letter response: "Thanks for submitting a report to the Reddit admin team. After investigating, we’ve found that the reported content doesn’t violate Reddit’s Content Policy." The absentee management of these platforms has turned them into an attractive nuisance. Getting more reports won't address the underlying issues.

OneLeggedCat 5y ago

Non-consensual tracking, on the other hand...

verisimi 5y ago

Come on, let's not beat around the bush. The answer to this is that we need to log on using our biomedical-crypto-citizen-id, just to go online. Everything we do _needs_ to be fully tracked. We need to serve our governments all the information about everyone, so that they can catch the baddies* - its essential! It really is true - someone did something bad online, so that justifies the loss of liberty for everyone! Get with it. Roll over. Right?!? * exceptions apply for key governmental employees.

croon 5y ago
You kid, but the dystopia of 1984 is probably pretty close to the utopia where everyones rights are protected while also being held accountable by a benevolent technocracy. I just understand that the latter won't last a day before it's abused and devolves into the former.
ydlr 5y ago
> * exceptions apply for key governmental employees. If government employees are exempt, where would they get the evidence needed to blackmail each other into maintaining the system.

tzs 5y ago

I sometimes think that we might need to approach this from the opposite direction. Declare a national "post your privates" day where we all post to all our social media pictures pictures of ourselves nude and/or engaging in sexual activity. Make this a regular event. Try to make it something everyone participates in. Fat people. Skinny people. People who climb on rocks. Tough people. Sissy people. Even people with chicken pox. Elderly people. Kids (but just nudes, nothing sexual). Singles. Couples. Whole families. Outside of the "post your junk" days social media would still ban such material. The idea is to make it so that if someone then does post intimate pictures of you without your permission the impact would be about as bad as someone posting an unauthorized picture of you having lunch. That wouldn't completely defang such photos of course, because sometimes the harm from the photo is not that it shows your private parts or shows you having sex but rather who it shows you with when you were doing it. A photo of Juliet with Romeo would have caused her trouble even if they were fully clothed and just talking. But for the cases where what makes the images damaging is the nudity or sex itself, normalizing this could help.

kelseyfrog 5y ago
This is like posting your SSN to make yourself immune to identity theft.

huetius 5y ago

Honest, if difficult, questions: Is there any benefit to legalized pornography that outweighs the horror of things like this? Does the amount of bureaucratic scaffolding and invasive surveillance by corporate and government actors required to police the boundary of consensual and non-consensual pornography leave you substantially more free than just punishing platforms which permit the dissemination of pornographic images?

dcallies 5y ago

Hey everyone! I’m a software engineer at Meta, and I helped work on the program that the newsroom post is about. Sorry I’m late, I only just found the thread! I might be able to answer some questions about the technology - many of the building blocks are open source, like the PDQ photo hashing algorithm (https://github.com/facebook/ThreatExchange/tree/main/pdq https://github.com/facebook/ThreatExchange/tree/main/pdq) mentioned in FAQ (https://stopncii.org/faq/ https://stopncii.org/faq/). I might not be able to answer every question, but I’ll do my best!