5 ms·

The false positive rate for any given image is not 1 in a trillion. Perceptual hashing just does not work like that. It also suffers from the birthday paradox p

The false positive rate for any given image is not 1 in a trillion. Perceptual hashing just does not work like that. It also suffers from the birthday paradox problem - as the database expands, and the total number of pictures expands, collisions become more likely. The parent poster does make the mistake of assuming that other pictures of kids will likely cause false positives. Anything could trigger a false positive - especially flesh tones. Like, say, the naughty pictures you've been taking of your (consenting) adult partner. I'm sure Apple's outsourced low-wage-country verification team will enjoy those.

kristofferR 5y ago
> Like, say, the naughty pictures you've been taking of your (consenting) adult partner. I'm sure Apple's outsourced low-wage-country verification team will enjoy those. I'm not sure they'll be able to after looking at CSAM all day...
- ajsnigrutin 5y ago
  Depends... This would be a great job for an actual pedophile.
windexh8er 5y ago
> The false positive rate for any given image is not 1 in a trillion. Perceptual hashing just does not work like that. It also suffers from the birthday paradox problem - as the database expands, and the total number of pictures expands, collisions become more likely. There was a good article [0] that was on HN a couple days ago that touches on the flat out lie regarding "one in a trillion" and how PhotoDNA sounds poorly thought out. [0] https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
- jrek 5y ago
  Apple is not using PhotoDNA, but in any event - this is a good article but they misconstrued the '1 in a trillion' quote, as is canvassed in the comments for the article itself. According to apple there is a 1 in a trillion chance of wrongly flagging an account, not a 1 in a trillion false positive rate for individual images, and we know that step to banning include: - multiple flagged images; and - human review The details for those processes hasn't been fully disclosed, and it isn't possible to say whether 1 in a trillion is a reasonable estimate or otherwise.
  - stickfigure 5y ago
    I don't want my private pictures reviewed by humans. What's the probability of that?
  - formerly_proven 5y ago
    "Don't worry, even if the system flags your baby pictures, we'll just hand them to the lowest-bidding subcontractor who will review the pictures of your naked baby (you know, to catch pedophiles!) and then probably not report you to the authorities. If this happens too often we're just going to ban you tho, this process costs money y'know?"
noobermin 5y ago
People really don't know about the birthday paradox, don't they? Should be required reading any time someone claims a false positive rate is "1 in a trillion."
FabHK 5y ago
> Anything could trigger a false positive Don't you think Apple has independently arrived at that very same possibility? Maybe even thought about it and extensively tested it? Then put a probability on it? And then chosen the threshold such that the overall probability of falsely flagging an account is, indeed, minuscule?
- thebooktocome 5y ago
  Given what I know about how FAANGs are run, they probably shrugged and wrote down whatever threshold felt right.
nonbirithm 5y ago
Then where are the news reports or articles of these false positives that would have shown up within the past decade? That's how long these companies have been using PhotoDNA on the server side. And the version of PhotoDNA from ten years ago would probably have been inferior to the version in place now. Is there even a single verifiable report of such a false positive? I feel that with the amount of attention brought to this issue, if there was such a report then it probably would have been brought up by now. Beyond that, assume that a false positive occurs and the innocent person is taken to court. Why would they have to fear being convicted if they don't actually hold any incriminating evidence? At most, that would become evidence against using perceptual hashing in future court cases. The issue in that case is the violation of the innocent person's privacy, not that they have a risk of being falsely convicted. The courts would still need admissible evidence, and I don't believe that only having a perceptual hash and a set of legally photographed images clears that bar. However, it becomes a completely separate issue if the false positives are "coincidentally" used to persecute marginalized groups in other countries where the same set of laws don't apply. But Apple has stated that they have no intention of expanding the system's scope to follow those laws. There isn't any evidence yet that Apple will do such a thing, or that they've already done it in the past. We are free to disbelieve them, but that's what they've stated. We can only hope that they won't change their minds.
- thebooktocome 5y ago
  > Why would they have to fear being convicted if they don't actually hold any incriminating evidence? Because going to court is extremely expensive? Because retaining legal council is extremely expensive? Because your district attorney will likely inflate the charges to coerce you into taking a plea deal, despite your innocence? Because you don't want all of your private documents, including ones completely unrelated to the alleged crime, entered into the court records? Because a "jury of your peers" can be convinced to believe just about anything? Because even if you're acquitted, most people will still believe you're guilty, and treat you as such? There are a myriad of reasons one shouldn't let the police go on mass criminal fishing expeditions. A lot of innocent people take collateral damage as a direct result of the incentives involved in an adversarial justice system.
nomel 5y ago
My guess is that the one in a trillion figure includes the threshold that must be exceeded, requiring multiple false positives, bringing the full chance of your account being flagged to 1 in a trillion.
- simondotau 5y ago
  Correct, the "1 in 1 trillion" does factor in the requirement for multiple images to match. From Apple's technical summary: "Using another technology called threshold secret sharing, the system ensures that the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images." "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match..." And when the manual review process sees that the images flagged aren't NCMEC classification A1 (A=prepubescent, 1=sex acts) the flag is cleared.
Grustaf 5y ago
> The false positive rate for any given image is not 1 in a trillion What's relevant is the overall false positive rate. If they require 6 matches for example, it's enough for each match to have a 1% false positive rate in order to get 1 in trillion overall.
- fxtentacle 5y ago
  Sadly not, because like the original comment implies, a false positive on a given image might cause it to be uploaded to Apple for manual review. So if Apple has a "1 in a trillion" 1e^-12 chance of flagging a person and they require (my guess) 3 hits to flag someone, then that would mean the chance of each single image being false positive is (1e^-12)^(1/3) = 1e^-4. So that means: Expect 1 in 1000 images to be uploaded to Apple for potential human verification.
  - tpush 5y ago
    No, manual review only happens after the threshold is reached. Meaning, there is a 1 in a trillion chance that threshold-number of images are manually verified by Apple.

kristofferR 5y ago

> Like, say, the naughty pictures you've been taking of your (consenting) adult partner. I'm sure Apple's outsourced low-wage-country verification team will enjoy those. I'm not sure they'll be able to after looking at CSAM all day...

ajsnigrutin 5y ago
Depends... This would be a great job for an actual pedophile.

windexh8er 5y ago

> The false positive rate for any given image is not 1 in a trillion. Perceptual hashing just does not work like that. It also suffers from the birthday paradox problem - as the database expands, and the total number of pictures expands, collisions become more likely. There was a good article [0] that was on HN a couple days ago that touches on the flat out lie regarding "one in a trillion" and how PhotoDNA sounds poorly thought out. [0] https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...

jrek 5y ago
Apple is not using PhotoDNA, but in any event - this is a good article but they misconstrued the '1 in a trillion' quote, as is canvassed in the comments for the article itself. According to apple there is a 1 in a trillion chance of wrongly flagging an account, not a 1 in a trillion false positive rate for individual images, and we know that step to banning include: - multiple flagged images; and - human review The details for those processes hasn't been fully disclosed, and it isn't possible to say whether 1 in a trillion is a reasonable estimate or otherwise.
- stickfigure 5y ago
  I don't want my private pictures reviewed by humans. What's the probability of that?
- formerly_proven 5y ago
  "Don't worry, even if the system flags your baby pictures, we'll just hand them to the lowest-bidding subcontractor who will review the pictures of your naked baby (you know, to catch pedophiles!) and then probably not report you to the authorities. If this happens too often we're just going to ban you tho, this process costs money y'know?"

noobermin 5y ago

People really don't know about the birthday paradox, don't they? Should be required reading any time someone claims a false positive rate is "1 in a trillion."

FabHK 5y ago

> Anything could trigger a false positive Don't you think Apple has independently arrived at that very same possibility? Maybe even thought about it and extensively tested it? Then put a probability on it? And then chosen the threshold such that the overall probability of falsely flagging an account is, indeed, minuscule?

thebooktocome 5y ago
Given what I know about how FAANGs are run, they probably shrugged and wrote down whatever threshold felt right.

nonbirithm 5y ago

Then where are the news reports or articles of these false positives that would have shown up within the past decade? That's how long these companies have been using PhotoDNA on the server side. And the version of PhotoDNA from ten years ago would probably have been inferior to the version in place now. Is there even a single verifiable report of such a false positive? I feel that with the amount of attention brought to this issue, if there was such a report then it probably would have been brought up by now. Beyond that, assume that a false positive occurs and the innocent person is taken to court. Why would they have to fear being convicted if they don't actually hold any incriminating evidence? At most, that would become evidence against using perceptual hashing in future court cases. The issue in that case is the violation of the innocent person's privacy, not that they have a risk of being falsely convicted. The courts would still need admissible evidence, and I don't believe that only having a perceptual hash and a set of legally photographed images clears that bar. However, it becomes a completely separate issue if the false positives are "coincidentally" used to persecute marginalized groups in other countries where the same set of laws don't apply. But Apple has stated that they have no intention of expanding the system's scope to follow those laws. There isn't any evidence yet that Apple will do such a thing, or that they've already done it in the past. We are free to disbelieve them, but that's what they've stated. We can only hope that they won't change their minds.

thebooktocome 5y ago
> Why would they have to fear being convicted if they don't actually hold any incriminating evidence? Because going to court is extremely expensive? Because retaining legal council is extremely expensive? Because your district attorney will likely inflate the charges to coerce you into taking a plea deal, despite your innocence? Because you don't want all of your private documents, including ones completely unrelated to the alleged crime, entered into the court records? Because a "jury of your peers" can be convinced to believe just about anything? Because even if you're acquitted, most people will still believe you're guilty, and treat you as such? There are a myriad of reasons one shouldn't let the police go on mass criminal fishing expeditions. A lot of innocent people take collateral damage as a direct result of the incentives involved in an adversarial justice system.

nomel 5y ago

My guess is that the one in a trillion figure includes the threshold that must be exceeded, requiring multiple false positives, bringing the full chance of your account being flagged to 1 in a trillion.

simondotau 5y ago
Correct, the "1 in 1 trillion" does factor in the requirement for multiple images to match. From Apple's technical summary: "Using another technology called threshold secret sharing, the system ensures that the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images." "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account. This is further mitigated by a manual review process wherein Apple reviews each report to confirm there is a match..." And when the manual review process sees that the images flagged aren't NCMEC classification A1 (A=prepubescent, 1=sex acts) the flag is cleared.

Grustaf 5y ago

> The false positive rate for any given image is not 1 in a trillion What's relevant is the overall false positive rate. If they require 6 matches for example, it's enough for each match to have a 1% false positive rate in order to get 1 in trillion overall.

fxtentacle 5y ago
Sadly not, because like the original comment implies, a false positive on a given image might cause it to be uploaded to Apple for manual review. So if Apple has a "1 in a trillion" 1e^-12 chance of flagging a person and they require (my guess) 3 hits to flag someone, then that would mean the chance of each single image being false positive is (1e^-12)^(1/3) = 1e^-4. So that means: Expect 1 in 1000 images to be uploaded to Apple for potential human verification.
- tpush 5y ago
  No, manual review only happens after the threshold is reached. Meaning, there is a 1 in a trillion chance that threshold-number of images are manually verified by Apple.