7 ms·

So the person would have to accept and save an image that when looks enough like CSAM to confuse a reviewer…

the8472 5y ago
Depending on what algorithm apple uses to generate the "sample" that's shown to the reviewer it may be possible to generate a large image that looks innocent unless downscaled with that specific algorithm and to a specific resolution
formerly_proven 5y ago
Yes, the diligent review performed by the lowest-bidding subcontractor is an excellent defense against career-ending criminal accusations. Nothing can go wrong, this is fine.
- saynay 5y ago
  I would think there is way easier ways to frame someone with CSAM then this. Like dump a thumbdrive of the stuff on them and report them to the police.
  - Sebb767 5y ago
    The police will not investigate every hint and a thumbdrive still has some plausible deniability. Evidence on your phone looks far worse and, thanks to this new process, law enforcement will receive actual evidence instead of just a hint.
hdjjhhvvhga 5y ago
My WhatsApp automatically saves all images to my photo roll. It has to be explicitly turned off. When the default is on, it's enough that the image is received and the victim has CP on their phone. After the initial shock they delete it, but the image has already been sent to Apple, where a reviewer marked it as CP. Since the user already gave them their full address data in order to be able to use the app store, Appla can automatically send a report to the police.
- zimpenfish 5y ago
  > but the image has already been sent to Apple, where a reviewer marked it as CP No, the images are only decryptable after a threshold (which appears to be about 30) is breached. If you've received 30 pieces of CSAM from WhatsApp contacts without blocking them and/or stopping WhatsApp from automatically saving to iCloud, I gotta say, it's on you at that point.
  - BrianOnHN 5y ago
    Victim blaming because of failure to meet some seemingly arbitrary limit, ok.
  - avianlyric 5y ago
    You’re aware that people sleep at night, and phones for the most part don’t, right?
  - basilgohar 5y ago
    Just a side point, a single WhatsApp message can contain up to 30 images. 30 is the literal max of a single message. So ONE MESSAGE could theoretically contain enough images to trip this threshold.
    - zimpenfish 5y ago
      > a single WhatsApp message can contain up to 30 images A fair point, yes, and somewhat scuppering towards my argument.
- tpush 5y ago
  > [...] Appla can automatically send a report to the police. Just to clarify, Apple doesn't report anyone to the police. They report to NCMEC, who presumably contacts law enforcement.
  - dannyw 5y ago
    FBI agents work for NCMEC. NCMEC is created by legislation. They ARE law enforcement, disguised as a non-profit.
avianlyric 5y ago
The reviewer may not be looking at the original image. But rather the visual derivative created during the hashing process and sent as part of the safety voucher. In this scenario you could create an image that looks like anything, but where it’s visual derivative is CSAM material. Currently iCloud isn’t encrypted, so Apple could just look at the original image. But in future is iCloud becomes encrypted, then the reporting will be don’t entirely based on the visual derivative. Although Apple could change this by include a unique crypto key for each uploaded images within their inner safety voucher, allowing them to decrypt images that match for the review process.
goohle 5y ago
Like this one: https://flickr.com/photos/tonysanchez/2526100122 https://flickr.com/photos/tonysanchez/2526100122
- tubbs 5y ago
  I would suggest not clicking this link on a work device.
kleene_op 5y ago
Not necessarily. If you know the method used by Apple to scale down flagged images before they are sent for review, you can make it so the scaled down version of the image shows a different, potentially misleading one instead: https://thume.ca/projects/2012/11/14/magic-png-files/ https://thume.ca/projects/2012/11/14/magic-png-files/ At the end of the day: - You can trick the user into saving an innocent looking image - You can trick Apple NN hashing function with a purposely generated hash - You can trick the reviewer with an explicit thumbnail There is no limit to how devilish one can be.
Grustaf 5y ago
Not one image. Ten, or maybe 50, who knows what the threshold is.
labcomputer 5y ago
So here's something I find interesting about this whole discussion: Everyone seems to assume the reviewers are honest actors. It occurs to me that compromising an already-hired reviewer (either through blackmail or bribery) or even just planting your own insider on the review team might not be that difficult. In fact, if your threat model includes nation-state adversaries, it seems crazy not to consider compromised reviewers. How hard would it really be for the CIA or NSA to get a few of their (under cover) people on the review team?
spoonjim 5y ago
I don't see how a perfectly legal and normal explicit photograph of someone's 20-year-old wife would be indistinguishable to an Apple reviewer from CSAM, especially since some people look much younger or much older than their chronological age. So first, there would be the horrendous breach of privacy for an Apple goon to be looking at this picture in the first place, which the person in the photograph never consented to, and second, could put the couple in legal hot water for absolutely no reason.
- brokenmachine 5y ago
  The personal photo is unlikely to match a photo in the CSAM database though, or at least that's what is claimed by Apple with no way to verify if it's true or not.

Depending on what algorithm apple uses to generate the "sample" that's shown to the reviewer it may be possible to generate a large image that looks innocent unless downscaled with that specific algorithm and to a specific resolution

formerly_proven 5y ago

Yes, the diligent review performed by the lowest-bidding subcontractor is an excellent defense against career-ending criminal accusations. Nothing can go wrong, this is fine.

saynay 5y ago
I would think there is way easier ways to frame someone with CSAM then this. Like dump a thumbdrive of the stuff on them and report them to the police.
- Sebb767 5y ago
  The police will not investigate every hint and a thumbdrive still has some plausible deniability. Evidence on your phone looks far worse and, thanks to this new process, law enforcement will receive actual evidence instead of just a hint.

hdjjhhvvhga 5y ago

My WhatsApp automatically saves all images to my photo roll. It has to be explicitly turned off. When the default is on, it's enough that the image is received and the victim has CP on their phone. After the initial shock they delete it, but the image has already been sent to Apple, where a reviewer marked it as CP. Since the user already gave them their full address data in order to be able to use the app store, Appla can automatically send a report to the police.

zimpenfish 5y ago
> but the image has already been sent to Apple, where a reviewer marked it as CP No, the images are only decryptable after a threshold (which appears to be about 30) is breached. If you've received 30 pieces of CSAM from WhatsApp contacts without blocking them and/or stopping WhatsApp from automatically saving to iCloud, I gotta say, it's on you at that point.
- BrianOnHN 5y ago
  Victim blaming because of failure to meet some seemingly arbitrary limit, ok.
- avianlyric 5y ago
  You’re aware that people sleep at night, and phones for the most part don’t, right?
- basilgohar 5y ago
  Just a side point, a single WhatsApp message can contain up to 30 images. 30 is the literal max of a single message. So ONE MESSAGE could theoretically contain enough images to trip this threshold.
  - zimpenfish 5y ago
    > a single WhatsApp message can contain up to 30 images A fair point, yes, and somewhat scuppering towards my argument.
tpush 5y ago
> [...] Appla can automatically send a report to the police. Just to clarify, Apple doesn't report anyone to the police. They report to NCMEC, who presumably contacts law enforcement.
- dannyw 5y ago
  FBI agents work for NCMEC. NCMEC is created by legislation. They ARE law enforcement, disguised as a non-profit.

avianlyric 5y ago

The reviewer may not be looking at the original image. But rather the visual derivative created during the hashing process and sent as part of the safety voucher. In this scenario you could create an image that looks like anything, but where it’s visual derivative is CSAM material. Currently iCloud isn’t encrypted, so Apple could just look at the original image. But in future is iCloud becomes encrypted, then the reporting will be don’t entirely based on the visual derivative. Although Apple could change this by include a unique crypto key for each uploaded images within their inner safety voucher, allowing them to decrypt images that match for the review process.

goohle 5y ago

Like this one: https://flickr.com/photos/tonysanchez/2526100122 https://flickr.com/photos/tonysanchez/2526100122

tubbs 5y ago
I would suggest not clicking this link on a work device.

kleene_op 5y ago

Not necessarily. If you know the method used by Apple to scale down flagged images before they are sent for review, you can make it so the scaled down version of the image shows a different, potentially misleading one instead: https://thume.ca/projects/2012/11/14/magic-png-files/ https://thume.ca/projects/2012/11/14/magic-png-files/ At the end of the day: - You can trick the user into saving an innocent looking image - You can trick Apple NN hashing function with a purposely generated hash - You can trick the reviewer with an explicit thumbnail There is no limit to how devilish one can be.

Grustaf 5y ago

Not one image. Ten, or maybe 50, who knows what the threshold is.

labcomputer 5y ago

So here's something I find interesting about this whole discussion: Everyone seems to assume the reviewers are honest actors. It occurs to me that compromising an already-hired reviewer (either through blackmail or bribery) or even just planting your own insider on the review team might not be that difficult. In fact, if your threat model includes nation-state adversaries, it seems crazy not to consider compromised reviewers. How hard would it really be for the CIA or NSA to get a few of their (under cover) people on the review team?

spoonjim 5y ago

I don't see how a perfectly legal and normal explicit photograph of someone's 20-year-old wife would be indistinguishable to an Apple reviewer from CSAM, especially since some people look much younger or much older than their chronological age. So first, there would be the horrendous breach of privacy for an Apple goon to be looking at this picture in the first place, which the person in the photograph never consented to, and second, could put the couple in legal hot water for absolutely no reason.

brokenmachine 5y ago
The personal photo is unlikely to match a photo in the CSAM database though, or at least that's what is claimed by Apple with no way to verify if it's true or not.