6 ms·

One additional issue that I haven't really seen discussed is how to handle a situation when a false accusation is made. If a person knows the right people who

One additional issue that I haven't really seen discussed is how to handle a situation when a false accusation is made. If a person knows the right people who work at these companies, things get sorted out, but I imagine sometimes a person is forced to just handle the consequences. Stepping away from CSAM and going back to something like developer account and apps getting banned on platforms for violating vague "guidelines". It's someone's livelihood that's sometimes destroyed. Demonetization, apps getting banned, payment processors freezing accounts are mostly black box events and most situations aren't even related to crimes dealing with CSAM. If it was something the government made a mistake with, there's legal ways to fight for your rights. There's generally a level of transparency that is afforded to you. It is concerning that people flagged for handling CSAM will not know if they have been manually reviewed. The need to keep the forwarding to authorities a secret is understandable, but a human review before forwarding is only necessary if you expect false positives to begin with. Keeping that flag secret seems like another black box you can't fight as a user. I don't deny the value of catching these criminals, but it throws the idea of due process out the window when the only assurance so far has been "trust us to do the right thing". It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. I am glad this hasn't flown under the radar just because it is Apple who is making these promises. I have heard non-tech people talk about this but there's a lot of misunderstanding.

hirvi74 5y ago
> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. So, I am not much of conspiracy theorist, but I do like to sometimes fantasize about alternative realities in which they were true. I am not saying the US government had any involvement in Apple's decision, but what if they did? I do agree with your point about how this topic more or less came out of Left-field. It's clear that Apple did not just recently acquire the technological ability to produce this feature in 2021. This feature could have been implemented years ago (like many other companies with a consumer-available cloud storage model already did to some degree). I am just curious if Apple did not really have a "choice" in this matter. Perhaps my monkey brain just want this to be the case.
- knaik94 5y ago
  I don't know for sure if the US government has publicly endorsed creating a backdoor after the San Bernardino shooting incident. That was 2015/2016. The Apple vs Epic case made an interesting email public about Apple's head of fraud acknowledging the problem of CSAM on their platform in Feb 2020 [1] but I agree with you that this kind of feature has to have been in development since before then because it's clearly been red teamed. The feature wasn't released until they knew they would have a response to all of the technical questions concerning the actual implementation. The UK policy, from 2017, around age verification [2] before giving access to pornographic material has definitely also played a part. Even youtube recently has made a strong verification effort going as far as asking for a credit card or photo ID as a response to COPPA [3] and EU policy. But Apple hasn't framed the porn blurring for minors feature as a response to that policy, which is surprising. And none of those government policies explain the need to do on device scanning of CSAM instead of in the cloud like everyone else. I personally believe Apple felt a requirement from a business perspective more than to avoid government regulation. It doesn't seem to be a secret warrant type situation, because publicly disclosure wouldn't really make sense. If catching criminals consuming/sharing CSAM is the motive, warning those criminals that this change is happening before implementation seems counter-productive. They even went as far as going after leakers a couple of months ago [4]. > The letter purportedly cautioned leakers that they must not disclose information about unreleased Apple projects because it may give Apple's competitors valuable information and "mislead customers, because what is disclosed may not be accurate." It's clear in hindsight that these "features" leaking early is what Apple was afraid of as it's been confirmed that the CSAM scanning algorithm was found, inactive, in iOS 14.3. The current stance is that Apple will have an additional check but if the presence of the scanning algorithm was leaked then, they wouldn't have been able to do the same PR spinning that they're doing now. I agree with you that it really doesn't fit Apple's narrative of privacy first. They are the same company that developed the security enclave to make a more secure product starting with the iphone 5s in 2013. I hope we do get some clarity on the motivation because it's clear that no one buys the "we did it for better privacy" narrative they are currently pushing. Their hand being forced doesn't seem out of the question to me either, but their own public response seems to make it doubtful to me. I think it speaks volumes that the first item that shows up on google when you search the term "icloud scanning apple" is a lifehacker article titled "How to Stop Apple From Scanning Your iPhone Photos Before iOS 15 Arrives". 1. https://www.theverge.com/22611236/epic-v-apple-emails-project-liberty-app-store-schiller-sweeney-cook-jobs#:~:text=71.%20February%202020%3A%20Apple%E2%80%99s%20head%20of%20fraud%20suggests%20Apple%20may%20be%20unwittingly%20providing%20%E2%80%9Cthe%20greatest%20platform%20for%20distributing%20child%20porn.%E2%80%9D https://www.theverge.com/22611236/epic-v-apple-emails-projec... 2. https://www.theguardian.com/society/2021/may/05/uk-government-faces-action-over-lack-of-age-checks-on-pornography-websites https://www.theguardian.com/society/2021/may/05/uk-governmen... 3. https://www.androidpolice.com/2021/06/24/why-youtube-wants-some-age-verification-from-you-all-of-the-sudden/ https://www.androidpolice.com/2021/06/24/why-youtube-wants-s... 4. https://www.macrumors.com/2021/06/24/reliable-leaker-kang-hit-with-warning/ https://www.macrumors.com/2021/06/24/reliable-leaker-kang-hi...
skinkestek 5y ago
> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. Not so weird actually in my opinion: Apple wants to make sure their customers doesn't get into trouble for no good reason, that would be bad for business. Personally for me, a European living in Norway I do have great respect for the local police (mostly). Great people, mostly going out of their way to serve the public (again, mostly). Norwegian child protection however is someone I'd rather stay clear of. They are well known to simultaneously ignore kids who need help and also harass innocent parents. Again, not all of them are like this, many of them try to do good, but they seem to a large extent to be working outside of control of the courts so if they mess up you have to go to a EU court to fix it. (Two or three cases just last 18 months from a small country like Norway.) So something similar might also be at play, but I don't know what reputation NCMEC has, only that it is well known that a number of people have gotten serious trouble because of overeager employees at photo labs reporting innocent images.
- knaik94 5y ago
  I honestly believe that's not Apple's place. Apple is a technology company, not a non-profit founded to help find missing or abused children. NCMEC has a pretty good and non-controversial reputation and you really only heard about it if you were working in that kind of law enforcement field. Or you knew of the database if you were personally working on tech for filtering and managing submission of photo/video from the public, like facebook image uploads and had to manage submitting reports of found CSAM. I personally only knew of the amber alerts, which are national broadcast messages that show up on my phone informing of cases of serious abduction incidents. I think for people informed about tech, the most unpopular thing about NCMEC is their stance on promoting a backdoor in all E2E encryption to ensure CSAM is flagged. [1] But even that, you understand where they're coming from even if you believe that backdoors are a bad idea. The NCMEC is a government funded non-profit for the explicit purpose of finding missing and exploited children. It's just not Apple's place unless it will also have a department for the explicit purpose of finding missing and exploited children. And at that point it's a branch of NCMEC working in Apple. Child protective services isn't NCMEC. And NCMEC isn't law enforcement, the organization works with law enforcement and doesn't directly go after anyone. I have never heard of people that have gotten into serious trouble because of overeager employees. I wasn't able to find it via google either, I would love to get more information if you have a link. I have heard of three letter government agencies allegedly planting CSAM on targets they want to get leverage on. But whether actual CSAM is planted or they lie in order to get probable cause to do searches and intimidate, I'm not sure. This article from 2019 outlines an instance where law enforcement accused someone of handling CSAM based on filenames and an IP address, but weren't able to find the file after doing an actual search on the device. [2] They ended up dropping the case instead of letting the lawyers examine the method used tag alleged predators. They actually dropped cases instead of disclosing how the software worked multiple times for the sake of protecting how the software works, and arguably let alleged predators walk free. But all of that is separate from NCMEC which just handles a cyber tipline and follows up on reports from tech companies about any potential CSAM and is in charge of handling the database of known positives in the US. Apple has stated they will require the database to be cross-checked across different countries and jurisdictions in order to ensure no one can add false matches. So they don't want to identify CSAM for just the US, they want it to be a global program even if they are just rolling it out in the US. 1. https://www.missingkids.org/theissues/end-to-end-encryption https://www.missingkids.org/theissues/end-to-end-encryption 2. https://www.propublica.org/article/prosecutors-dropping-child-porn-charges-after-software-tools-are-questioned https://www.propublica.org/article/prosecutors-dropping-chil...
ec109685 5y ago
There’s a human review at other cloud providers too.
- knaik94 5y ago
  From what I've read about the other cloud providers, they make it easy to report matches to NCMEC but I haven't seen anything about a human review as a middle step. I really only know of PhotoDNA which it seems like Apple essentially has copied but not identically. PhotoDNA does not include human review. From what I understand, legally, only people at NCMEC are allowed to view the material raw which is what's confusing about Apple's announcement of human review. https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna
  - ectopod 5y ago
    Exactly. I do not believe that Apple will be reviewing the actual images. More likely they will be reviewing the metadata for "chain of evidence" purposes. When Apple pass evidence to NCMEC they will be able to say "this was flagged for CSAM and we have double-checked that it actually came from this person's phone".
    - ec109685 5y ago
      No, they will be reviewing a version of the image. They say it explicitly in their design documents.
      - ectopod 5y ago
        A version of the image that has had the picture data removed or obscured (so only the metadata can be reviewed) is consistent with that statement. This is how marketing works. A company comes up with a plan and then the marketers come up with the best sounding technically true description consistent with it. The fact that the description creates an entirely false impression on the reader is by design.
dwaite 5y ago
> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. Apple seems to design systems like this so that there are multiple parties involved, in order to require coordination to abuse the system and to prevent attempted abuses of the system from becoming public knowledge. Until independent review, Apple/reviewers has not seen if the input hash actually corresponds to CSAM.
gadders 5y ago
Obviously you need to post here and once the issue gets on the front page of HN, someone at Apple will see it and fix it.

hirvi74 5y ago

> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. So, I am not much of conspiracy theorist, but I do like to sometimes fantasize about alternative realities in which they were true. I am not saying the US government had any involvement in Apple's decision, but what if they did? I do agree with your point about how this topic more or less came out of Left-field. It's clear that Apple did not just recently acquire the technological ability to produce this feature in 2021. This feature could have been implemented years ago (like many other companies with a consumer-available cloud storage model already did to some degree). I am just curious if Apple did not really have a "choice" in this matter. Perhaps my monkey brain just want this to be the case.

knaik94 5y ago
I don't know for sure if the US government has publicly endorsed creating a backdoor after the San Bernardino shooting incident. That was 2015/2016. The Apple vs Epic case made an interesting email public about Apple's head of fraud acknowledging the problem of CSAM on their platform in Feb 2020 [1] but I agree with you that this kind of feature has to have been in development since before then because it's clearly been red teamed. The feature wasn't released until they knew they would have a response to all of the technical questions concerning the actual implementation. The UK policy, from 2017, around age verification [2] before giving access to pornographic material has definitely also played a part. Even youtube recently has made a strong verification effort going as far as asking for a credit card or photo ID as a response to COPPA [3] and EU policy. But Apple hasn't framed the porn blurring for minors feature as a response to that policy, which is surprising. And none of those government policies explain the need to do on device scanning of CSAM instead of in the cloud like everyone else. I personally believe Apple felt a requirement from a business perspective more than to avoid government regulation. It doesn't seem to be a secret warrant type situation, because publicly disclosure wouldn't really make sense. If catching criminals consuming/sharing CSAM is the motive, warning those criminals that this change is happening before implementation seems counter-productive. They even went as far as going after leakers a couple of months ago [4]. > The letter purportedly cautioned leakers that they must not disclose information about unreleased Apple projects because it may give Apple's competitors valuable information and "mislead customers, because what is disclosed may not be accurate." It's clear in hindsight that these "features" leaking early is what Apple was afraid of as it's been confirmed that the CSAM scanning algorithm was found, inactive, in iOS 14.3. The current stance is that Apple will have an additional check but if the presence of the scanning algorithm was leaked then, they wouldn't have been able to do the same PR spinning that they're doing now. I agree with you that it really doesn't fit Apple's narrative of privacy first. They are the same company that developed the security enclave to make a more secure product starting with the iphone 5s in 2013. I hope we do get some clarity on the motivation because it's clear that no one buys the "we did it for better privacy" narrative they are currently pushing. Their hand being forced doesn't seem out of the question to me either, but their own public response seems to make it doubtful to me. I think it speaks volumes that the first item that shows up on google when you search the term "icloud scanning apple" is a lifehacker article titled "How to Stop Apple From Scanning Your iPhone Photos Before iOS 15 Arrives". 1. https://www.theverge.com/22611236/epic-v-apple-emails-project-liberty-app-store-schiller-sweeney-cook-jobs#:~:text=71.%20February%202020%3A%20Apple%E2%80%99s%20head%20of%20fraud%20suggests%20Apple%20may%20be%20unwittingly%20providing%20%E2%80%9Cthe%20greatest%20platform%20for%20distributing%20child%20porn.%E2%80%9D https://www.theverge.com/22611236/epic-v-apple-emails-projec... 2. https://www.theguardian.com/society/2021/may/05/uk-government-faces-action-over-lack-of-age-checks-on-pornography-websites https://www.theguardian.com/society/2021/may/05/uk-governmen... 3. https://www.androidpolice.com/2021/06/24/why-youtube-wants-some-age-verification-from-you-all-of-the-sudden/ https://www.androidpolice.com/2021/06/24/why-youtube-wants-s... 4. https://www.macrumors.com/2021/06/24/reliable-leaker-kang-hit-with-warning/ https://www.macrumors.com/2021/06/24/reliable-leaker-kang-hi...

skinkestek 5y ago

> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. Not so weird actually in my opinion: Apple wants to make sure their customers doesn't get into trouble for no good reason, that would be bad for business. Personally for me, a European living in Norway I do have great respect for the local police (mostly). Great people, mostly going out of their way to serve the public (again, mostly). Norwegian child protection however is someone I'd rather stay clear of. They are well known to simultaneously ignore kids who need help and also harass innocent parents. Again, not all of them are like this, many of them try to do good, but they seem to a large extent to be working outside of control of the courts so if they mess up you have to go to a EU court to fix it. (Two or three cases just last 18 months from a small country like Norway.) So something similar might also be at play, but I don't know what reputation NCMEC has, only that it is well known that a number of people have gotten serious trouble because of overeager employees at photo labs reporting innocent images.

knaik94 5y ago
I honestly believe that's not Apple's place. Apple is a technology company, not a non-profit founded to help find missing or abused children. NCMEC has a pretty good and non-controversial reputation and you really only heard about it if you were working in that kind of law enforcement field. Or you knew of the database if you were personally working on tech for filtering and managing submission of photo/video from the public, like facebook image uploads and had to manage submitting reports of found CSAM. I personally only knew of the amber alerts, which are national broadcast messages that show up on my phone informing of cases of serious abduction incidents. I think for people informed about tech, the most unpopular thing about NCMEC is their stance on promoting a backdoor in all E2E encryption to ensure CSAM is flagged. [1] But even that, you understand where they're coming from even if you believe that backdoors are a bad idea. The NCMEC is a government funded non-profit for the explicit purpose of finding missing and exploited children. It's just not Apple's place unless it will also have a department for the explicit purpose of finding missing and exploited children. And at that point it's a branch of NCMEC working in Apple. Child protective services isn't NCMEC. And NCMEC isn't law enforcement, the organization works with law enforcement and doesn't directly go after anyone. I have never heard of people that have gotten into serious trouble because of overeager employees. I wasn't able to find it via google either, I would love to get more information if you have a link. I have heard of three letter government agencies allegedly planting CSAM on targets they want to get leverage on. But whether actual CSAM is planted or they lie in order to get probable cause to do searches and intimidate, I'm not sure. This article from 2019 outlines an instance where law enforcement accused someone of handling CSAM based on filenames and an IP address, but weren't able to find the file after doing an actual search on the device. [2] They ended up dropping the case instead of letting the lawyers examine the method used tag alleged predators. They actually dropped cases instead of disclosing how the software worked multiple times for the sake of protecting how the software works, and arguably let alleged predators walk free. But all of that is separate from NCMEC which just handles a cyber tipline and follows up on reports from tech companies about any potential CSAM and is in charge of handling the database of known positives in the US. Apple has stated they will require the database to be cross-checked across different countries and jurisdictions in order to ensure no one can add false matches. So they don't want to identify CSAM for just the US, they want it to be a global program even if they are just rolling it out in the US. 1. https://www.missingkids.org/theissues/end-to-end-encryption https://www.missingkids.org/theissues/end-to-end-encryption 2. https://www.propublica.org/article/prosecutors-dropping-child-porn-charges-after-software-tools-are-questioned https://www.propublica.org/article/prosecutors-dropping-chil...

ec109685 5y ago

There’s a human review at other cloud providers too.

knaik94 5y ago
From what I've read about the other cloud providers, they make it easy to report matches to NCMEC but I haven't seen anything about a human review as a middle step. I really only know of PhotoDNA which it seems like Apple essentially has copied but not identically. PhotoDNA does not include human review. From what I understand, legally, only people at NCMEC are allowed to view the material raw which is what's confusing about Apple's announcement of human review. https://www.microsoft.com/en-us/photodna https://www.microsoft.com/en-us/photodna
- ectopod 5y ago
  Exactly. I do not believe that Apple will be reviewing the actual images. More likely they will be reviewing the metadata for "chain of evidence" purposes. When Apple pass evidence to NCMEC they will be able to say "this was flagged for CSAM and we have double-checked that it actually came from this person's phone".
  - ec109685 5y ago
    No, they will be reviewing a version of the image. They say it explicitly in their design documents.
    - ectopod 5y ago
      A version of the image that has had the picture data removed or obscured (so only the metadata can be reviewed) is consistent with that statement. This is how marketing works. A company comes up with a plan and then the marketers come up with the best sounding technically true description consistent with it. The fact that the description creates an entirely false impression on the reader is by design.

dwaite 5y ago

> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. Apple seems to design systems like this so that there are multiple parties involved, in order to require coordination to abuse the system and to prevent attempted abuses of the system from becoming public knowledge. Until independent review, Apple/reviewers has not seen if the input hash actually corresponds to CSAM.

gadders 5y ago

Obviously you need to post here and once the issue gets on the front page of HN, someone at Apple will see it and fix it.