3 ms·

> The second protection [against mis-inclusion of non CSAM hashes] is human review: there is no automated reporting in Apple’s system. All positive matches must

> The second protection [against mis-inclusion of non CSAM hashes] is human review: there is no automated reporting in Apple’s system. All positive matches must be visually confirmed by Apple as containing CSAM before Apple will disable the account and file a report with the child safety organization. I don't understand this at all. As I understand it, part of the problem is that — in the US — Apple isn't legally allowed to review, transmit or do anything else with suspected CSAM images, so they can't have a manual review process (or even check that their neural hashing is working as expected on the real dataset). Does anyone else have any idea of what this is trying to describe? If Apple really are somehow reviewing flagged photos to confirm that they're CSAM and not maliciously flagged files before sending any reports, then that does make the system substantially more resilient to Five Eyes abuse (not that I wish that job on anyone). Edit: There's more context later in the document > First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation. This is more confusing: visually comparison using a second perceptual hash doesn't actually to provide any protection against mis-inclusion of non-CSAM images: it just double-checks that the image really was a match in the database (ie. protects against hash-collision errors), but it does't check that the database contained actual CSAM. Apple explicitly says that this process protects against mis-inclusion though, which doesn't make sense to me yet.

shuckles 5y ago
Your understanding is incorrect. Apple can, and is in fact required to, verify that they have actual CSAM before forwarding it to the Cyber Tip line. At that point, they must delete the information within 60 days.
- drodgers 5y ago
  Interesting. In that case, do you know why they talk about reviews only seeing "visual derivatives" (from the second perceptual hash)? Either these 'derivatives' basically contain the original image (so reviewers can verify that it's actual CSAM) and there's no point in using derivatives at all, or they're more abstract (eg. 8x8 pixellated images) in which case the reviewer can't see the actual content (but could confirm a database match). Edit: I was able to find the answer, they suggest that the 'visual derivative' is something like a 'low-resolution version' of the original image, so the content should still be clearly visible.
  - shuckles 5y ago
    Apple claims that “data minimization” is one of their privacy pillars, and this was probably an attempt at that. You could imagine an inverted colors or lower res image gets the point across without subjecting you or Apple to all the high fidelity images you have in your library.
- heavyset_go 5y ago
  > Apple can, and is in fact required to, verify that they have actual CSAM It's not Apple's job to decide if something is CSAM or not. They're required to report it if they suspect it is, even if they can't confirm it.
FabHK 5y ago
Your phone transmits the images with their security envelope (which was computed on device and contains neural hash and "visual derivative") to the iCloud server. During that process, Apple does not know whether there's any CSAM in it, so they can transmit legally. Then the server determines whether the number of matches exceeds the threshold. Only if that is the case (by crypto magic) can the security envelope of the flagged images only (by crypto magic) be unlocked, and the "visual derivative" be reviewed. (Note that if (at a later stage) E2EE is enabled for the photos, the images themselves would never be accessible by Apple or LE, whether flagged or not, if I understand the design correctly).
- dwaite 5y ago
  Your understanding seems correct. After a positive evaluation from Apple, the CyberTipline report is filed to NCMEC, which operates as a clearinghouse and notifies law enforcement. Law enforcement then gets a court order, which will cause Apple to release requested/available information about that account. If photos later are outside the key escrow system, Apple would not be able to release the photos encryption keys, and would only be able to share 'public' photo share information which the user has opted to share without encryption to make it web-accessible. Presumably in this case, the visual derivatives would still be used as evidence, but there would be 5th amendment arguments around forcing access to a broader set of photos.
zionic 5y ago
It’s simple, the reviewers will be cops (so viewing CP is legal). They will pass on/hit report at rates that make a FISA judge blush.

shuckles 5y ago

Your understanding is incorrect. Apple can, and is in fact required to, verify that they have actual CSAM before forwarding it to the Cyber Tip line. At that point, they must delete the information within 60 days.

drodgers 5y ago
Interesting. In that case, do you know why they talk about reviews only seeing "visual derivatives" (from the second perceptual hash)? Either these 'derivatives' basically contain the original image (so reviewers can verify that it's actual CSAM) and there's no point in using derivatives at all, or they're more abstract (eg. 8x8 pixellated images) in which case the reviewer can't see the actual content (but could confirm a database match). Edit: I was able to find the answer, they suggest that the 'visual derivative' is something like a 'low-resolution version' of the original image, so the content should still be clearly visible.
- shuckles 5y ago
  Apple claims that “data minimization” is one of their privacy pillars, and this was probably an attempt at that. You could imagine an inverted colors or lower res image gets the point across without subjecting you or Apple to all the high fidelity images you have in your library.
heavyset_go 5y ago
> Apple can, and is in fact required to, verify that they have actual CSAM It's not Apple's job to decide if something is CSAM or not. They're required to report it if they suspect it is, even if they can't confirm it.

FabHK 5y ago

Your phone transmits the images with their security envelope (which was computed on device and contains neural hash and "visual derivative") to the iCloud server. During that process, Apple does not know whether there's any CSAM in it, so they can transmit legally. Then the server determines whether the number of matches exceeds the threshold. Only if that is the case (by crypto magic) can the security envelope of the flagged images only (by crypto magic) be unlocked, and the "visual derivative" be reviewed. (Note that if (at a later stage) E2EE is enabled for the photos, the images themselves would never be accessible by Apple or LE, whether flagged or not, if I understand the design correctly).

dwaite 5y ago
Your understanding seems correct. After a positive evaluation from Apple, the CyberTipline report is filed to NCMEC, which operates as a clearinghouse and notifies law enforcement. Law enforcement then gets a court order, which will cause Apple to release requested/available information about that account. If photos later are outside the key escrow system, Apple would not be able to release the photos encryption keys, and would only be able to share 'public' photo share information which the user has opted to share without encryption to make it web-accessible. Presumably in this case, the visual derivatives would still be used as evidence, but there would be 5th amendment arguments around forcing access to a broader set of photos.

zionic 5y ago

It’s simple, the reviewers will be cops (so viewing CP is legal). They will pass on/hit report at rates that make a FISA judge blush.