4 ms·

> and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the databa

> and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. I don't think NCMEC adds random images they find to the A1 list without knowing their origin. > until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Who is this an attack on? A moderation contractor maybe, but not the protester. "Further scrutiny" doesn't mean "you go to jail", it means someone looks at the pictures.

matthewdgreen 3y ago
NCMEC’s database is smaller and infrequently updated, but Facebook’s database is much larger and more actively updated. This is also one of the reasons that Facebook/Meta produces many more reports than other providers who use the NCMEC database. Governments view the limited reach of the NCMEC database as a problem, and are encouraging providers to use more complete databases and to develop tools to detect novel CSAM (this is included in the EU proposed regulations.) “Further scrutiny” here means your images are reported to a provider, which (at that point) means your provider has a list of reports that defeat the guarantees of E2E encryption. The threat model looks something like this: https://www.justice.gov/opa/pr/former-twitter-employee-found-guilty-acting-agent-foreign-government-and-unlawfully-sharing https://www.justice.gov/opa/pr/former-twitter-employee-found...
LorenPechtel 3y ago
Image B' *is* CSAM. A competent authority will add it when it comes to their attention.
LorenPechtel 3y ago
Look at some of the cases in the news--the tech giants are more interested in ensuring that bad guys don't use their systems than in justice. Remember that case not too long ago with a telehealth appointment, they sent a picture of something on their 2? year old's penis to the doc, asking if it was an issue. The police cleared him, but he's forever guilty in Google's eyes.
- astrange 3y ago
  Google was not required to implement that the way they did, which is basically a server-side scanning policy of having contractors look at your nudes if you put them on your cloud drive. NCMEC may approve of it, sure, but another aspect of their not being a government agency is that you don't have to listen to them.
  - tuhriel 3y ago
    Problem is, as they are not a goverment agency and don't really see you as customer (more like a product) you have almost no avenue to get your account reinstated... And as there are some people (and even companies) who rely on google services and their account there it can really shoot you in the knee
    - astrange 3y ago
      "Not being a government agency" was about NCMEC, not Google.
Hizonner 3y ago
It means your account gets a higher risk score, which may mean it gets given a "timeout", gets downranked in "the algorithm", gets outright shadowbanned, or may even get completely shut off. All in a completely automated way.
- astrange 3y ago
  Nobody is "downranking users in the algorithm" because they think the user /is a pedophile/. That is mixing up social media controversies. Your account could certainly be locked until someone looks at it though, yes.
  - tuhriel 3y ago
    Yes, and we all know how easy it is to get a real person from the big companies to take a look at something... Or even reach one
Retr0id 3y ago
Regarding the first part, if that really is the relevant policy, fair enough. I don't know the specific policies used to include or exclude a given image. Where can I find the details? I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards. The reason apple's threshold exists in the first place is because individual false-positives happen. Some of the images leaked from the victim's device may be entirely unrelated. In the specific example of an environmental protester, there might even be images documenting "crimes" (entirely unrelated to CSAM), due to increased criminalization of protest techniques. A system that may be manipulated (anonymously, from great distance) to trigger spot-checks on the devices of anyone I don't like is a broken system.
- astrange 3y ago
  > Regarding the first part, if that really is the relevant policy, fair enough. I don't know the specific policies used to include or exclude a given image. Where can I find the details? I'm not sure either or I would've linked it. I have implemented such a reporting system though (compliant to US law, which is privacy-preserving relatively speaking, not any upcoming EU laws, which like all other EU law seems like a huge pain to live under.) > I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards. To be clear, this is only if you're using a cloud storage service like iCloud Photos, Google Drive etc. It's meant to be a strict improvement over the usual setup, which is that your data isn't hidden from the cloud provider at all and they can just look at whatever. It would certainly be had to have any scanning if you're not opting into a cloud service.
chislobog 3y ago
The proposed legislation and rules indicate that the “EU Centre,” Interpol, and local law enforcement and governments would all have access to alerts (reports, unfiltered) and the databases and systems used to generate them.

matthewdgreen 3y ago

NCMEC’s database is smaller and infrequently updated, but Facebook’s database is much larger and more actively updated. This is also one of the reasons that Facebook/Meta produces many more reports than other providers who use the NCMEC database. Governments view the limited reach of the NCMEC database as a problem, and are encouraging providers to use more complete databases and to develop tools to detect novel CSAM (this is included in the EU proposed regulations.) “Further scrutiny” here means your images are reported to a provider, which (at that point) means your provider has a list of reports that defeat the guarantees of E2E encryption. The threat model looks something like this: https://www.justice.gov/opa/pr/former-twitter-employee-found-guilty-acting-agent-foreign-government-and-unlawfully-sharing https://www.justice.gov/opa/pr/former-twitter-employee-found...

LorenPechtel 3y ago

Image B' *is* CSAM. A competent authority will add it when it comes to their attention.

LorenPechtel 3y ago

Look at some of the cases in the news--the tech giants are more interested in ensuring that bad guys don't use their systems than in justice. Remember that case not too long ago with a telehealth appointment, they sent a picture of something on their 2? year old's penis to the doc, asking if it was an issue. The police cleared him, but he's forever guilty in Google's eyes.

astrange 3y ago
Google was not required to implement that the way they did, which is basically a server-side scanning policy of having contractors look at your nudes if you put them on your cloud drive. NCMEC may approve of it, sure, but another aspect of their not being a government agency is that you don't have to listen to them.
- tuhriel 3y ago
  Problem is, as they are not a goverment agency and don't really see you as customer (more like a product) you have almost no avenue to get your account reinstated... And as there are some people (and even companies) who rely on google services and their account there it can really shoot you in the knee
  - astrange 3y ago
    "Not being a government agency" was about NCMEC, not Google.

Hizonner 3y ago

It means your account gets a higher risk score, which may mean it gets given a "timeout", gets downranked in "the algorithm", gets outright shadowbanned, or may even get completely shut off. All in a completely automated way.

astrange 3y ago
Nobody is "downranking users in the algorithm" because they think the user /is a pedophile/. That is mixing up social media controversies. Your account could certainly be locked until someone looks at it though, yes.
- tuhriel 3y ago
  Yes, and we all know how easy it is to get a real person from the big companies to take a look at something... Or even reach one

Retr0id 3y ago

Regarding the first part, if that really is the relevant policy, fair enough. I don't know the specific policies used to include or exclude a given image. Where can I find the details? I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards. The reason apple's threshold exists in the first place is because individual false-positives happen. Some of the images leaked from the victim's device may be entirely unrelated. In the specific example of an environmental protester, there might even be images documenting "crimes" (entirely unrelated to CSAM), due to increased criminalization of protest techniques. A system that may be manipulated (anonymously, from great distance) to trigger spot-checks on the devices of anyone I don't like is a broken system.

astrange 3y ago
> Regarding the first part, if that really is the relevant policy, fair enough. I don't know the specific policies used to include or exclude a given image. Where can I find the details? I'm not sure either or I would've linked it. I have implemented such a reporting system though (compliant to US law, which is privacy-preserving relatively speaking, not any upcoming EU laws, which like all other EU law seems like a huge pain to live under.) > I consider someone looking at pictures from my local device without my consent to be an attack on my privacy, regardless of the content, or whether they send me to jail afterwards. To be clear, this is only if you're using a cloud storage service like iCloud Photos, Google Drive etc. It's meant to be a strict improvement over the usual setup, which is that your data isn't hidden from the cloud provider at all and they can just look at whatever. It would certainly be had to have any scanning if you're not opting into a cloud service.

chislobog 3y ago

The proposed legislation and rules indicate that the “EU Centre,” Interpol, and local law enforcement and governments would all have access to alerts (reports, unfiltered) and the databases and systems used to generate them.