5 ms·

None of those attacks would work against the system as described by Apple. The only photos scanned are items in your photo library prior to upload to iCloud. Yo

None of those attacks would work against the system as described by Apple. The only photos scanned are items in your photo library prior to upload to iCloud. Your browser cache is not scanned. Hash collisions would fail human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.

soziawa 5y ago
> Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM. The whitepapers provided by Apple do not say what the human reviews consists of. They could just look at the hashes to make sure there isn‘t a bug in their system.
- simondotau 5y ago
  > The whitepapers provided by Apple do not say what the human reviews consists of. At minimum what we know is that each flagged image generates a "safety voucher" which consists of metadata, plus a low-resolution greyscale version of the image. The human review process involves viewing the metadata and thumbnail content enclosed in each safety voucher which cumulatively caused that account to be flagged.
  - foobar33333 5y ago
    A human at Apple likely doesn't get access to anything. I assume it would be part of the police group under strict restrictions checking these.
    - simondotau 5y ago
      The data is not sent to a "police group", it is sent to NCMEC. From Apple's FAQ: Will CSAM detection in iCloud Photos falsely flag innocent people to law enforcement? No. The system is designed to be very accurate, and the likelihood that the system would incorrectly flag any given account is less than one in one trillion per year. In addition, any time an account is flagged by the system, Apple conducts human review before making a report to NCMEC. As a result, system errors or attacks will not result in innocent people being reported to NCMEC.
      - douglasisshiny 5y ago
        NCMEC then makes those images available to the appropriate law enforcement agency after the fact.
        simondotau 5y ago
        Yes, if they're CSAM.
deleted 5y ago
[deleted]
T3OU-736 5y ago
> Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM. I truly wish I could subscribe to this optimistic view. Experience tends to show this to be unlikely. Two factors combine against it: 1. There is no negative consequence for a mis-flag (to the reviewer) 2. This set up is a tool, and like many tools, inventive humans will find a way to subvert it in the name of convenience. I am referring to NSLs from U.S. Patriot Act as an example. Since CSAM is such a toxic thing (let's stipulate that CSAM itself is unequivocally bad), there is less tendency to examine it closely for, well, CSAM-ness.
- simondotau 5y ago
  Again, I'm only pointing out how this conflicts with Apple's description of their system. I'm in no position to know whether their description is accurate or how it will actually operate in the real world. For the sake of argument, let's assume you're correct and Apple's review team are lazy shits who don't look at the images. Okay, so Apple then sends the report onto NCMEC. What are they going to do when they open the report and it turned out the images Apple reported were hash collisions?
  - lvh 5y ago
    My understanding (from someone who would know but said this in a Chatham House rules space) is that NCMEC is already incredibly underfunded, understaffed, and backlogged. Similar incentives apply to them. They're a nonprofit: a private organization who has significantly fewer dollars than Apple does.
    - simondotau 5y ago
      The critical follow-up question is what do NCMEC do with their backlog? Unless they're dumping this backlog directly at the feet of law enforcement, I don't see how this changes the equation.
  - 5y ago
GistNoesis 5y ago
One obvious problem with human review is steganography. The picture can look normal to the human eye, but if it contains hidden content (in the least significant bit of each pixel for example so that the hash is unchanged), a forensic software will definitely notice, raise some flags, and extract the hidden offensive content automatically, leaving the human reviewer no other choice but to report you. If Apple says they are not going to look for hidden content, then they are just handling a free pass which render the whole scanning thing pointless.
- madmoose 5y ago
  What you're talking about here has nothing to do with what Apple is implementing.
- simondotau 5y ago
  I'm confused what scenario you're positing here. Given the widespread adoption of encrypted communications, steganography is of no use to traffickers of CSAM. Steganography generally serves only one purpose, which is to transfer material in public view with plausible deniabilty—such as leaking material out of a military facility which has exceedingly robust data protection processes. Apple have explicitly said that their hash algorithm is only concerned with visible elements of the image.
  - GistNoesis 5y ago
    I'm speaking about the adversarial scenario of an attacker trying to frame a target. He just need to get on your phone an image with hidden content that has a hash collision with the database. Traffickers and consumers of CSAM know that their content is illegal to possess and store so they sometime use steganography software to store the offensive data inside their innocuous photo library. This way when they can browse their private collection via the lens of the steganography software and they don't have some suspicious encrypted file that would attract attention of someone they share the computer with.
    - simondotau 5y ago
veidr 5y ago
> Hash collisions would fail human review. This (pervasive, over the past couple days) idea that Apple (of all major tech companies, lol!) will be capable of manually reviewing tens of thousands of automated detections per day is... nuts. The "system as described by Apple" doesn't comport to reality, because it relies on human review. If you remove the human review, the system is fucked. But no company on the planet has the capability to sanely and ethically (to say nothing of competently or effectively) conduct such review, at the scale of iOS.
- simondotau 5y ago
  Tens of thousands of automated detections per day? Unlikely. More likely tens per year. Remember, this isn't a porn detector combined with a child detector. It is hashing images in your cloud-enabled photo library and comparing those to hashes of images already known to child abuse authorities. In addition, consider how monumentally unlikely it is for any CSAM enthusiast to copy these illicit photos into their phone's general camera roll alongside pictures of their family and dog. This is only going to catch the stupidest and sloppiest CSAM enthusiast.
  - detaro 5y ago
    For comparison to your "likely tens per year" number, Facebook is running the same kind of detectors and reports ~20 million instances a year: https://twitter.com/durumcrustulum/status/1423776278847451138 https://twitter.com/durumcrustulum/status/142377627884745113...
    - simondotau 5y ago
      That's a summary number of many kinds of reports, of which CSAM hash matches would be one part. That summary number also includes accusations of child sex trafficking and online enticement. I wouldn't be surprised if reported allegations of trafficking and enticement were in excess of 99.9% of Facebook's reporting. But since they don't break it out, I can only guess. Given that guesses aren't useful to anyone, it would be interesting if you know of any statistics from any of the major tech vendors, of the reporting frequency of just CSAM hash matches.
jann 5y ago
Messaging apps like WhatsApp will save to your photo library though (unless disabled). So any photo sent to you would be scanned. If you someone sent you a bunch of files, that might trigger a manual review, that would most likely flag your account. I wouldn't expect that immediately deleting them would stop the review process.
- jbverschoor 5y ago
  again, I hope someone sends a couple of executives the recently posted images, to make a point
  - themaninthedark 5y ago
    That is why they talk about having a manual review process. So that when someone wealthy or politically connected triggers the system there is a review.
- simondotau 5y ago
  I haven't used WhatsApp, but I'm tempted to call bullshit on that. I've never used any messaging app on iOS which saves photos to your photo library. Doing so would make no sense and would surely be infuriating. It's also worth noting that apps on iOS can't save to your photo library unless you give them explicit permission.
  - tpush 5y ago
    WhatsApp does by default save received images to your photo library (as opposed to e.g. iMessage). You can turn that off, though. And the permission to read from a user's photo library (to e.g. post images) includes the ability to write to it.
    - simondotau 5y ago
      Gross. I can't fathom how anyone would put up with that.
  - 5y ago
alpaca128 5y ago
> Hash collisions would fail human review You mean like the absolutely perfect human review of appstore content that's known for both false positives and false negatives? Neither automatic nor manual (human) review works 100% reliably. And believing otherwise will only ruin lives.
- cmsj 5y ago
  You are absolutely correct that neither automatic nor manual review is ever going to be 100% accurate. I would like to believe though that for this system to fully fail an innocent person, the following would all need to have failed: 1) Coincidental CSAM hash collision 2) Incorrect manual review by Apple 3) Incorrect subsequent review by NCMEC 4) Inability of a lawyer to obtain the original image for presentation during a trial/appeal which seems kind of unlikely? (although it's certainly the case that once steps 1, 2 and 3 have failed, the person's reputation is likely damaged even if they are able to prove their innocence in court). The wider question here is, should 100% accuracy be the bar by which we judge this? I don't think we expect the law enforcement system to be 100% right, hence principles like the presumption of innocence and right to appeal, and even then it gets things wrong sometimes.
  - alpaca128 5y ago
    There are known cases of police faking AI-generated evidence[0]. There's no reason why Apple would be immune against such things. And the recent British post office scandal shows that even without manipulation false faith in technology as evidence can destroy hundreds of lives. The low chance of an error going through that whole chain of checks also increases the trust in that system even in the case of a false positive. And all this is assuming it will never be expanded from CSAM to other content. Apple is already rolling out a censored version of iOS in China. [0] https://www.vice.com/en/article/qj8xbq/police-are-telling-shotspotter-to-alter-evidence-from-gunshot-detecting-ai https://www.vice.com/en/article/qj8xbq/police-are-telling-sh...
  - samrolken 5y ago
    You’re missing the threshold that is part of this system. You would need multiple hash collisions across multiple photos to trigger these mechanisms.
zionic 5y ago
This is out of date. It took less than a week for Apple to announce this tech is coming to "3rd party apps".
themaninthedark 5y ago
It depends on how the attack was crafted: Step 1: Get copies of pictures of targets kid in bath from phone/SNS Step 2: Manipulate pictures so that hash collides with CSAM Step 3: Get pictures back on targets phone so they get scanned. If it were me, I would try and get a series of photos from the target, and manipulate several that look most borderline. That way it looks like more than a one off. Now if there is an Apple review, the person who views them will see some suspect pictures and would confirm. Now the target would have to get someone to review the original pictures vs the modified pictures. Good luck with the defense.

soziawa 5y ago

simondotau 5y ago
> The whitepapers provided by Apple do not say what the human reviews consists of. At minimum what we know is that each flagged image generates a "safety voucher" which consists of metadata, plus a low-resolution greyscale version of the image. The human review process involves viewing the metadata and thumbnail content enclosed in each safety voucher which cumulatively caused that account to be flagged.
- foobar33333 5y ago
  A human at Apple likely doesn't get access to anything. I assume it would be part of the police group under strict restrictions checking these.
  - simondotau 5y ago
    The data is not sent to a "police group", it is sent to NCMEC. From Apple's FAQ: Will CSAM detection in iCloud Photos falsely flag innocent people to law enforcement? No. The system is designed to be very accurate, and the likelihood that the system would incorrectly flag any given account is less than one in one trillion per year. In addition, any time an account is flagged by the system, Apple conducts human review before making a report to NCMEC. As a result, system errors or attacks will not result in innocent people being reported to NCMEC.
    - douglasisshiny 5y ago
      NCMEC then makes those images available to the appropriate law enforcement agency after the fact.
      - simondotau 5y ago
        Yes, if they're CSAM.

deleted 5y ago

[deleted]

T3OU-736 5y ago

> Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM. I truly wish I could subscribe to this optimistic view. Experience tends to show this to be unlikely. Two factors combine against it: 1. There is no negative consequence for a mis-flag (to the reviewer) 2. This set up is a tool, and like many tools, inventive humans will find a way to subvert it in the name of convenience. I am referring to NSLs from U.S. Patriot Act as an example. Since CSAM is such a toxic thing (let's stipulate that CSAM itself is unequivocally bad), there is less tendency to examine it closely for, well, CSAM-ness.

simondotau 5y ago
Again, I'm only pointing out how this conflicts with Apple's description of their system. I'm in no position to know whether their description is accurate or how it will actually operate in the real world. For the sake of argument, let's assume you're correct and Apple's review team are lazy shits who don't look at the images. Okay, so Apple then sends the report onto NCMEC. What are they going to do when they open the report and it turned out the images Apple reported were hash collisions?
- lvh 5y ago
  My understanding (from someone who would know but said this in a Chatham House rules space) is that NCMEC is already incredibly underfunded, understaffed, and backlogged. Similar incentives apply to them. They're a nonprofit: a private organization who has significantly fewer dollars than Apple does.
  - simondotau 5y ago
    The critical follow-up question is what do NCMEC do with their backlog? Unless they're dumping this backlog directly at the feet of law enforcement, I don't see how this changes the equation.
- 5y ago

GistNoesis 5y ago

One obvious problem with human review is steganography. The picture can look normal to the human eye, but if it contains hidden content (in the least significant bit of each pixel for example so that the hash is unchanged), a forensic software will definitely notice, raise some flags, and extract the hidden offensive content automatically, leaving the human reviewer no other choice but to report you. If Apple says they are not going to look for hidden content, then they are just handling a free pass which render the whole scanning thing pointless.

madmoose 5y ago
What you're talking about here has nothing to do with what Apple is implementing.
simondotau 5y ago
I'm confused what scenario you're positing here. Given the widespread adoption of encrypted communications, steganography is of no use to traffickers of CSAM. Steganography generally serves only one purpose, which is to transfer material in public view with plausible deniabilty—such as leaking material out of a military facility which has exceedingly robust data protection processes. Apple have explicitly said that their hash algorithm is only concerned with visible elements of the image.
- GistNoesis 5y ago
  I'm speaking about the adversarial scenario of an attacker trying to frame a target. He just need to get on your phone an image with hidden content that has a hash collision with the database. Traffickers and consumers of CSAM know that their content is illegal to possess and store so they sometime use steganography software to store the offensive data inside their innocuous photo library. This way when they can browse their private collection via the lens of the steganography software and they don't have some suspicious encrypted file that would attract attention of someone they share the computer with.
  - simondotau 5y ago

veidr 5y ago

> Hash collisions would fail human review. This (pervasive, over the past couple days) idea that Apple (of all major tech companies, lol!) will be capable of manually reviewing tens of thousands of automated detections per day is... nuts. The "system as described by Apple" doesn't comport to reality, because it relies on human review. If you remove the human review, the system is fucked. But no company on the planet has the capability to sanely and ethically (to say nothing of competently or effectively) conduct such review, at the scale of iOS.

simondotau 5y ago
Tens of thousands of automated detections per day? Unlikely. More likely tens per year. Remember, this isn't a porn detector combined with a child detector. It is hashing images in your cloud-enabled photo library and comparing those to hashes of images already known to child abuse authorities. In addition, consider how monumentally unlikely it is for any CSAM enthusiast to copy these illicit photos into their phone's general camera roll alongside pictures of their family and dog. This is only going to catch the stupidest and sloppiest CSAM enthusiast.
- detaro 5y ago
  For comparison to your "likely tens per year" number, Facebook is running the same kind of detectors and reports ~20 million instances a year: https://twitter.com/durumcrustulum/status/1423776278847451138 https://twitter.com/durumcrustulum/status/142377627884745113...
  - simondotau 5y ago
    That's a summary number of many kinds of reports, of which CSAM hash matches would be one part. That summary number also includes accusations of child sex trafficking and online enticement. I wouldn't be surprised if reported allegations of trafficking and enticement were in excess of 99.9% of Facebook's reporting. But since they don't break it out, I can only guess. Given that guesses aren't useful to anyone, it would be interesting if you know of any statistics from any of the major tech vendors, of the reporting frequency of just CSAM hash matches.

jann 5y ago

Messaging apps like WhatsApp will save to your photo library though (unless disabled). So any photo sent to you would be scanned. If you someone sent you a bunch of files, that might trigger a manual review, that would most likely flag your account. I wouldn't expect that immediately deleting them would stop the review process.

jbverschoor 5y ago
again, I hope someone sends a couple of executives the recently posted images, to make a point
- themaninthedark 5y ago
  That is why they talk about having a manual review process. So that when someone wealthy or politically connected triggers the system there is a review.
simondotau 5y ago
I haven't used WhatsApp, but I'm tempted to call bullshit on that. I've never used any messaging app on iOS which saves photos to your photo library. Doing so would make no sense and would surely be infuriating. It's also worth noting that apps on iOS can't save to your photo library unless you give them explicit permission.
- tpush 5y ago
  WhatsApp does by default save received images to your photo library (as opposed to e.g. iMessage). You can turn that off, though. And the permission to read from a user's photo library (to e.g. post images) includes the ability to write to it.
  - simondotau 5y ago
    Gross. I can't fathom how anyone would put up with that.
- 5y ago

alpaca128 5y ago

> Hash collisions would fail human review You mean like the absolutely perfect human review of appstore content that's known for both false positives and false negatives? Neither automatic nor manual (human) review works 100% reliably. And believing otherwise will only ruin lives.

cmsj 5y ago
You are absolutely correct that neither automatic nor manual review is ever going to be 100% accurate. I would like to believe though that for this system to fully fail an innocent person, the following would all need to have failed: 1) Coincidental CSAM hash collision 2) Incorrect manual review by Apple 3) Incorrect subsequent review by NCMEC 4) Inability of a lawyer to obtain the original image for presentation during a trial/appeal which seems kind of unlikely? (although it's certainly the case that once steps 1, 2 and 3 have failed, the person's reputation is likely damaged even if they are able to prove their innocence in court). The wider question here is, should 100% accuracy be the bar by which we judge this? I don't think we expect the law enforcement system to be 100% right, hence principles like the presumption of innocence and right to appeal, and even then it gets things wrong sometimes.
- alpaca128 5y ago
  There are known cases of police faking AI-generated evidence[0]. There's no reason why Apple would be immune against such things. And the recent British post office scandal shows that even without manipulation false faith in technology as evidence can destroy hundreds of lives. The low chance of an error going through that whole chain of checks also increases the trust in that system even in the case of a false positive. And all this is assuming it will never be expanded from CSAM to other content. Apple is already rolling out a censored version of iOS in China. [0] https://www.vice.com/en/article/qj8xbq/police-are-telling-shotspotter-to-alter-evidence-from-gunshot-detecting-ai https://www.vice.com/en/article/qj8xbq/police-are-telling-sh...
- samrolken 5y ago
  You’re missing the threshold that is part of this system. You would need multiple hash collisions across multiple photos to trigger these mechanisms.

zionic 5y ago

This is out of date. It took less than a week for Apple to announce this tech is coming to "3rd party apps".

themaninthedark 5y ago

It depends on how the attack was crafted: Step 1: Get copies of pictures of targets kid in bath from phone/SNS Step 2: Manipulate pictures so that hash collides with CSAM Step 3: Get pictures back on targets phone so they get scanned. If it were me, I would try and get a series of photos from the target, and manipulate several that look most borderline. That way it looks like more than a one off. Now if there is an Apple review, the person who views them will see some suspect pictures and would confirm. Now the target would have to get someone to review the original pictures vs the modified pictures. Good luck with the defense.