6 ms·

I’m sorry, at what point there was any confusion that Apple is going to use our phones to do scanning?

slg 5y ago
That is the where of the story. There was a inarguably confusion over the when, what, and how of the story. It was not universally understood that this would only apply to photos sent to iCloud. It was not universally understood that this was only looking for previously known CSAM. It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system. I understand if you consider the where more important than the others, but it is simply a fact that there was confusion on what exactly was happening here.
- lstodd 5y ago
  > some sort of hash matching so photos you took yourself would not trigger the system. This is ignorance in extreme.
- hypothesis 5y ago
  I appreciate your more detailed clarification and agree with ‘where’ conclusion. To the extent that other parts of this story was explained to us by Apple, I did try to clarify some exaggeration in other thread.
- asddubs 5y ago
  There's always going to be plenty of people commenting who didn't even bother to read the article at all. But by and large, from what I saw, people did understand the nuances of this and outlined how little stands in the way of expanding these policies' scope once the technology is in place
- cwkoss 5y ago
  I think "universally understood" is doing a lot of work to portray a much higher degree of certainty about each of those statements than is justified. A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan. "this would only apply to photos sent to iCloud." for now, until scope creeps. "this was only looking for previously known CSAM." for now, until scope creeps. "using some sort of hash matching so photos you took yourself would not trigger the system." well this one is immediately concerning even within claimed scope because there ARE going to be false positives that apple records some database. Millions of iphone users are going to have a non-zero "possible childporn" score. They are building an engine for iphone users to self-incriminate. If they rigidly hold the scope to only what they announced and never expand, it could be argued that this is a reasonable concession to fight CSAM. However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates.
  - slg 5y ago
    >A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan. The how helps show us how changing this system is not a subtle change. It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones. That would require a new system since the current one is only hash matching. >However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates. This is an arbitrary line that is being drawn. These are photos that are marked for sending to iCloud. Whether the scanning happens on the phone before they are sent or in the cloud after they sent is largely immaterial when it comes to the impact of the code. People are acting as if the line Apple drew was motivated by technology. That was never the deciding factor. Technology is the easy part here. That line was only a policy line and that policy has not changed. Only photos that are sent to iCloud are scanned. If you fear Apple changing that policy going forward, you should have always feared Apple changing that policy.
    - cwkoss 5y ago
      > It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones. This is a strawman. Identifying *novel* CSAM is a very hard problem to do accurately - the reason they can't flip the switch is because they don't have the technical capability. All of the other things people are concerned about are things that apple does have the capability to do. EDIT to reply since at max thread depth: *novel* image detection was never on the table, I think you missed that word > This is an arbitrary line that is being drawn. It seems the vast majority of people in this thread disagree that this line is arbitrary. EDIT: whether the threshold is "verging on impossible." depends entirely on the effective false positive rate. If apple's claimed 1 in 1 trillion rate is true, it's probably not a concern. However, I find it unlikely that perceptual hashes on portions of images won't have higher false positive rates when subject matter is similar (non-CSAM legal adult porn, or images of children in swimsuits, etc). If that rises to 1 in 1 million for these types of images, that's hundreds of thousands of people being falsely accused.
- insulanus 5y ago
  You forgot the who - Whose fault is it that those points were not clearly communicated? - Who wrote the perceptual hash matching code? - Who is allowed to audit the code, the review system, and the hash database? - Who updates this code? - Who decides if your phone OS is updated? - Who decides the iCloud upload defaults? - Who decides if you are reported? - Who asked for this feature?
  - romwell 5y ago
    - Who is going to have their dog killed by a misguided SWAT team that doesn't bother analyzing the automated report before acting on it?
- romwell 5y ago
  It was not universally "understood" because it was not universally agreed upon. To wit: * It was not universally understood that this would only apply to photos sent to iCloud. Since the scanning doesn't happen on iCloud, this distinction is irrelevant. "We are going to intrusively scan the subset of your photos that you care enough to back up to the cloud that we've been pushing to you for years" is pretty clear. * It was not universally understood that this was only looking for previously known CSAM. It was only looking for whatever is in an opaque database which, according to a third party we don't have any contract with, contains CSAM. * It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system. Yeah right, I feel totally safe knowing that I won't be falsely reported to FBI by a "some sort of" hash matching. Here's a hash function: f(x) = 0 for all x It's "some sort of" hash, too.
  - fingerlocks 5y ago
    > Since the scanning doesn't happen on iCloud, this distinction is irrelevant. It’s really not irrelevant. A third party photo library that avoids using the PhotoKit library would not be touched by the CSAM detector. There are many of these on the App Store. One step further and store the photos encrypted, with a custom renderer that decrypts the content on the heap, and that would take some tremendous performance-hitting detection abilities it’s extremely unlikely to ever happen.
    - romwell 5y ago
      "Look, there's a hack that allows you to avoid this at the cost of convenience that you bought the device for!" OK.
      - fingerlocks 5y ago
        “Don’t support this company and it’s entire product line because this one app is scanning for kiddie porn”. Misrepresenting each other’s arguments is not how you have a discussion.
- tomp 5y ago
  Your problem is trusting evil people at face value.
notriddle 5y ago
Nobody's confused about that. > HN is a tech forum and I still saw plenty of people here worried about how they would get in trouble for having innocent photos of their own children on their phone. They're confused about this. NeuralHash doesn't look for pictures of naked kids. It looks for pictures that are identical to the ones they've put in their signatures list. The problem is that Apple claims that the signatures in their list are all pictures of sexually-abused kids, but we have no way of verifying that. Heck, they don't even have any way of verifying that. Everyone just has to take NCMEC's word for it.
- lazide 5y ago
  You’re also wrong no? The perceptual hashing doesn’t match EXACTLY the same photos, it intentionally matches photos similar to the same photos, so a minor rotate by a few degrees or crop or whatever also matches
- cwkoss 5y ago
  Perceptual hashing can have collisions, and they will be at a higher rate than completely "identical". The public does not know what the false positive rate is for 'average iphone user pictures'. As engineers we can be certain the false positive rate is not zero. This means that some number of iphone users are going to have non zero "possible child pornographer" scores in the apple database. The false positive rate is crucial to understanding how concerning this should be. If the average iphone user has 1000 photos, and the false positive rate is the claimed 1 in 1 trillion, there is a 1 in a billion chance that you'll be flagged as a potential child pornographer. (~1 in the world will be falsely accused). This seems reasonable enough with the apple-internal screening step. If the chunking and perceptual hashing functionally ends up having a much higher false positive rate for images which have similarities to the dataset (parents' pictures of kids playing shirtless, legal adult porn, etc), the false positive rate could actually be more like 1 in 1 million or worse. In which case there are potentially hundreds of thousands of people who will be falsely accused by this system. How many matches will US judges require before they sign warrants for arrests, search and seizure of digital devices? If they are technically competent it shouldn't only be 1, but I don't trust all judges to understand probability well enough to require multiple matches.
benhurmarcel 5y ago
Why does it matter that much which CPU runs the check?

slg 5y ago

That is the where of the story. There was a inarguably confusion over the when, what, and how of the story. It was not universally understood that this would only apply to photos sent to iCloud. It was not universally understood that this was only looking for previously known CSAM. It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system. I understand if you consider the where more important than the others, but it is simply a fact that there was confusion on what exactly was happening here.

lstodd 5y ago
> some sort of hash matching so photos you took yourself would not trigger the system. This is ignorance in extreme.
hypothesis 5y ago
I appreciate your more detailed clarification and agree with ‘where’ conclusion. To the extent that other parts of this story was explained to us by Apple, I did try to clarify some exaggeration in other thread.
asddubs 5y ago
There's always going to be plenty of people commenting who didn't even bother to read the article at all. But by and large, from what I saw, people did understand the nuances of this and outlined how little stands in the way of expanding these policies' scope once the technology is in place
cwkoss 5y ago
I think "universally understood" is doing a lot of work to portray a much higher degree of certainty about each of those statements than is justified. A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan. "this would only apply to photos sent to iCloud." for now, until scope creeps. "this was only looking for previously known CSAM." for now, until scope creeps. "using some sort of hash matching so photos you took yourself would not trigger the system." well this one is immediately concerning even within claimed scope because there ARE going to be false positives that apple records some database. Millions of iphone users are going to have a non-zero "possible childporn" score. They are building an engine for iphone users to self-incriminate. If they rigidly hold the scope to only what they announced and never expand, it could be argued that this is a reasonable concession to fight CSAM. However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates.
- slg 5y ago
  >A lot of the contention wasn't about the specifics of their plan, but rather how subtle changes could vastly expand the scope of their plan. The how helps show us how changing this system is not a subtle change. It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones. That would require a new system since the current one is only hash matching. >However, in making the announcement, they boldly stepped past their existing hard line in privacy (local device content is private and not surveilled by apple), so it seems naive to expect that this announcement reflects the eventual scope of this self-incrimination engine for the next decade of apple updates. This is an arbitrary line that is being drawn. These are photos that are marked for sending to iCloud. Whether the scanning happens on the phone before they are sent or in the cloud after they sent is largely immaterial when it comes to the impact of the code. People are acting as if the line Apple drew was motivated by technology. That was never the deciding factor. Technology is the easy part here. That line was only a policy line and that policy has not changed. Only photos that are sent to iCloud are scanned. If you fear Apple changing that policy going forward, you should have always feared Apple changing that policy.
  - cwkoss 5y ago
    > It isn't like they can flip a switch and suddenly they are identifying new suspected CSAM on people's phones. This is a strawman. Identifying *novel* CSAM is a very hard problem to do accurately - the reason they can't flip the switch is because they don't have the technical capability. All of the other things people are concerned about are things that apple does have the capability to do. EDIT to reply since at max thread depth: *novel* image detection was never on the table, I think you missed that word > This is an arbitrary line that is being drawn. It seems the vast majority of people in this thread disagree that this line is arbitrary. EDIT: whether the threshold is "verging on impossible." depends entirely on the effective false positive rate. If apple's claimed 1 in 1 trillion rate is true, it's probably not a concern. However, I find it unlikely that perceptual hashes on portions of images won't have higher false positive rates when subject matter is similar (non-CSAM legal adult porn, or images of children in swimsuits, etc). If that rises to 1 in 1 million for these types of images, that's hundreds of thousands of people being falsely accused.
insulanus 5y ago
You forgot the who - Whose fault is it that those points were not clearly communicated? - Who wrote the perceptual hash matching code? - Who is allowed to audit the code, the review system, and the hash database? - Who updates this code? - Who decides if your phone OS is updated? - Who decides the iCloud upload defaults? - Who decides if you are reported? - Who asked for this feature?
- romwell 5y ago
  - Who is going to have their dog killed by a misguided SWAT team that doesn't bother analyzing the automated report before acting on it?
romwell 5y ago
It was not universally "understood" because it was not universally agreed upon. To wit: * It was not universally understood that this would only apply to photos sent to iCloud. Since the scanning doesn't happen on iCloud, this distinction is irrelevant. "We are going to intrusively scan the subset of your photos that you care enough to back up to the cloud that we've been pushing to you for years" is pretty clear. * It was not universally understood that this was only looking for previously known CSAM. It was only looking for whatever is in an opaque database which, according to a third party we don't have any contract with, contains CSAM. * It was not universally understood that they were using some sort of hash matching so photos you took yourself would not trigger the system. Yeah right, I feel totally safe knowing that I won't be falsely reported to FBI by a "some sort of" hash matching. Here's a hash function: f(x) = 0 for all x It's "some sort of" hash, too.
- fingerlocks 5y ago
  > Since the scanning doesn't happen on iCloud, this distinction is irrelevant. It’s really not irrelevant. A third party photo library that avoids using the PhotoKit library would not be touched by the CSAM detector. There are many of these on the App Store. One step further and store the photos encrypted, with a custom renderer that decrypts the content on the heap, and that would take some tremendous performance-hitting detection abilities it’s extremely unlikely to ever happen.
  - romwell 5y ago
    "Look, there's a hack that allows you to avoid this at the cost of convenience that you bought the device for!" OK.
    - fingerlocks 5y ago
      “Don’t support this company and it’s entire product line because this one app is scanning for kiddie porn”. Misrepresenting each other’s arguments is not how you have a discussion.
tomp 5y ago
Your problem is trusting evil people at face value.

notriddle 5y ago

Nobody's confused about that. > HN is a tech forum and I still saw plenty of people here worried about how they would get in trouble for having innocent photos of their own children on their phone. They're confused about this. NeuralHash doesn't look for pictures of naked kids. It looks for pictures that are identical to the ones they've put in their signatures list. The problem is that Apple claims that the signatures in their list are all pictures of sexually-abused kids, but we have no way of verifying that. Heck, they don't even have any way of verifying that. Everyone just has to take NCMEC's word for it.

lazide 5y ago
You’re also wrong no? The perceptual hashing doesn’t match EXACTLY the same photos, it intentionally matches photos similar to the same photos, so a minor rotate by a few degrees or crop or whatever also matches
cwkoss 5y ago
Perceptual hashing can have collisions, and they will be at a higher rate than completely "identical". The public does not know what the false positive rate is for 'average iphone user pictures'. As engineers we can be certain the false positive rate is not zero. This means that some number of iphone users are going to have non zero "possible child pornographer" scores in the apple database. The false positive rate is crucial to understanding how concerning this should be. If the average iphone user has 1000 photos, and the false positive rate is the claimed 1 in 1 trillion, there is a 1 in a billion chance that you'll be flagged as a potential child pornographer. (~1 in the world will be falsely accused). This seems reasonable enough with the apple-internal screening step. If the chunking and perceptual hashing functionally ends up having a much higher false positive rate for images which have similarities to the dataset (parents' pictures of kids playing shirtless, legal adult porn, etc), the false positive rate could actually be more like 1 in 1 million or worse. In which case there are potentially hundreds of thousands of people who will be falsely accused by this system. How many matches will US judges require before they sign warrants for arrests, search and seizure of digital devices? If they are technically competent it shouldn't only be 1, but I don't trust all judges to understand probability well enough to require multiple matches.

benhurmarcel 5y ago

Why does it matter that much which CPU runs the check?