4 ms·

The article considers "an entity that is allowed to propose new entries to the CSAM database". You don't even need this! You could target a whole "social clust

The article considers "an entity that is allowed to propose new entries to the CSAM database". You don't even need this! You could target a whole "social cluster" of people without having any special privileges within this system. As an example, lets say you want to attack environmental protesters. For image A, you create a meme about climate change. For image B, you procure something that looks, to humans, like CSAM (as described in the article). Craft B′ such that f(B′) = f(A) (also as described). Now, all you have to do is anonymously publish B′ to a platform that is actively moderated/monitored. Unfortunate users will see it and report it as CSAM, and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. Now you can start sending out image A, the meme, to your target demographic. You won't be able to post it on "mainstream" platforms with server-side fingerprint scanning, but there are plenty of other avenues for it to spread. If it's a good meme, it will propagate organically through group-chats and DMs, and eventually find its way onto devices with client-side scanning. Apple's proposed device scanning system had a threshold before your device would be flagged, so repeat all these steps a whole bunch of times, until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Unwitting victims who do try to post the meme to a mainstream platform with fingerprint matching may risk getting their accounts flagged and taken down, and they might have no way of knowing what triggered it. This would lead to, of course, automated censorship of environmental protest groups.

hju22_-3 3y ago
That's some next level shit. I dig it.
FireBeyond 3y ago
> Apple's proposed device scanning system had a threshold before your device would be flagged Which always struck me as odd as it would extremely easy to spin this as “Apple detected CSAM on this device but their policy is only to alert authorities once a set quantity of CSAM is found…”
- LorenPechtel 3y ago
  It means they recognize there can be false positives. And that you can be the innocent recipient of it.
  - FireBeyond 3y ago
    I absolutely understand that's what they're thinking. I also know it will only take one case of someone being arrested for a related crime, and material being found on their phone that's under the threshold for reporting, and the associated headlines that could be written.
astrange 3y ago
> and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. I don't think NCMEC adds random images they find to the A1 list without knowing their origin. > until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Who is this an attack on? A moderation contractor maybe, but not the protester. "Further scrutiny" doesn't mean "you go to jail", it means someone looks at the pictures.
- matthewdgreen 3y ago
  NCMEC’s database is smaller and infrequently updated, but Facebook’s database is much larger and more actively updated. This is also one of the reasons that Facebook/Meta produces many more reports than other providers who use the NCMEC database. Governments view the limited reach of the NCMEC database as a problem, and are encouraging providers to use more complete databases and to develop tools to detect novel CSAM (this is included in the EU proposed regulations.) “Further scrutiny” here means your images are reported to a provider, which (at that point) means your provider has a list of reports that defeat the guarantees of E2E encryption. The threat model looks something like this: https://www.justice.gov/opa/pr/former-twitter-employee-found-guilty-acting-agent-foreign-government-and-unlawfully-sharing https://www.justice.gov/opa/pr/former-twitter-employee-found...
- LorenPechtel 3y ago
  Image B' *is* CSAM. A competent authority will add it when it comes to their attention.
- LorenPechtel 3y ago
  Look at some of the cases in the news--the tech giants are more interested in ensuring that bad guys don't use their systems than in justice. Remember that case not too long ago with a telehealth appointment, they sent a picture of something on their 2? year old's penis to the doc, asking if it was an issue. The police cleared him, but he's forever guilty in Google's eyes.
  - 3y ago
logi_sko 3y ago
Even better: Distribute image A for a while before publishing B' and it can spread via all the server side scanning channels as well. Then when f(B')=f(A) is added to the database, people sse suddenly caught with apparently legal images.

hju22_-3 3y ago

That's some next level shit. I dig it.

FireBeyond 3y ago

> Apple's proposed device scanning system had a threshold before your device would be flagged Which always struck me as odd as it would extremely easy to spin this as “Apple detected CSAM on this device but their policy is only to alert authorities once a set quantity of CSAM is found…”

LorenPechtel 3y ago
It means they recognize there can be false positives. And that you can be the innocent recipient of it.
- FireBeyond 3y ago
  I absolutely understand that's what they're thinking. I also know it will only take one case of someone being arrested for a related crime, and material being found on their phone that's under the threshold for reporting, and the associated headlines that could be written.

astrange 3y ago

> and if the platform fulfills its obligations, that report will bubble up to the relevant authorities, who will review it and add its fingerprint to the database. I don't think NCMEC adds random images they find to the A1 list without knowing their origin. > until the average meme-savvy environmental protester's device gets flagged for further scrutiny. Who is this an attack on? A moderation contractor maybe, but not the protester. "Further scrutiny" doesn't mean "you go to jail", it means someone looks at the pictures.

matthewdgreen 3y ago
NCMEC’s database is smaller and infrequently updated, but Facebook’s database is much larger and more actively updated. This is also one of the reasons that Facebook/Meta produces many more reports than other providers who use the NCMEC database. Governments view the limited reach of the NCMEC database as a problem, and are encouraging providers to use more complete databases and to develop tools to detect novel CSAM (this is included in the EU proposed regulations.) “Further scrutiny” here means your images are reported to a provider, which (at that point) means your provider has a list of reports that defeat the guarantees of E2E encryption. The threat model looks something like this: https://www.justice.gov/opa/pr/former-twitter-employee-found-guilty-acting-agent-foreign-government-and-unlawfully-sharing https://www.justice.gov/opa/pr/former-twitter-employee-found...
LorenPechtel 3y ago
Image B' *is* CSAM. A competent authority will add it when it comes to their attention.
LorenPechtel 3y ago
Look at some of the cases in the news--the tech giants are more interested in ensuring that bad guys don't use their systems than in justice. Remember that case not too long ago with a telehealth appointment, they sent a picture of something on their 2? year old's penis to the doc, asking if it was an issue. The police cleared him, but he's forever guilty in Google's eyes.
- 3y ago

logi_sko 3y ago

Even better: Distribute image A for a while before publishing B' and it can spread via all the server side scanning channels as well. Then when f(B')=f(A) is added to the database, people sse suddenly caught with apparently legal images.