6 ms·

This is wrong - the iCloud check is against known CSAM hashes, the false positive rate is essentially zero.

by fossuser 5y ago

hhh 5y ago
One in one trillion chance per year, per the paper on the Apple site.
- fossuser 5y ago
  Yeah, that’s essentially zero.
  - deleted 5y ago
    [deleted]
- patmcc 5y ago
  I see the 'one in one trillion' but it's a bit vague; I read it as '1 in trillion chance' per image. So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.
  - jeromegv 5y ago
    They have also indicated that a single match won't be enough to trigger it, so an accidental match (being that 1 in a trillion person) is not enough.
- jhayward 5y ago
  I would like to bet Apple's market value against that number being correct in an adversarial context.
paul_f 5y ago
This seems naive, there are always false positives
- fossuser 5y ago
  It’s not naive, it’s math.
  - mpol 5y ago
    It's not a math problem, it's a human problem. suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.
    - fossuser 5y ago
      That isn’t CSAM. https://www.nytimes.com/interactive/2019/09/28/us/child-sex-abuse.html https://www.nytimes.com/interactive/2019/09/28/us/child-sex-... Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.
      - farmerstan 5y ago
        Who determines whether something is CSAM? How do you or I know that every single one of those images actually is CSAM? How do we know if the FBI or CIA or CCP adds hashes of innocent pics in order to pin a crime against someone?
- DSingularity 5y ago
  No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.
  - 5y ago
2pEXgD0fZ5cF 5y ago
> the false positive rate is essentially zero I highly doubt that
- DSingularity 5y ago
  Based on what? Unless you point out a flaw in the math, it’s zero.
  - 2pEXgD0fZ5cF 5y ago
    > Based on what? Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1] What we are left with is "trust in Apple" not "trust in math". [1]: https://news.ycombinator.com/item?id=27878333 https://news.ycombinator.com/item?id=27878333
bnj 5y ago
Yes but my understanding is that the hashes are perceptual, as opposed to cryptographic. If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate. And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.
- fossuser 5y ago
  I suspect that's why they have some threshold that moves the false positive rate to one in one trillion. The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.
  - bnj 5y ago
    This is a great point, thanks for adding that detail.
  - samatman 5y ago
    It was pretty dumb of Apple to announce both of these things at the same time. They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles. I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them. But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.
    - deleted 5y ago
      [deleted]
  - lamontcg 5y ago
    It doesn't matter if the cryptographic algorithm is one in a trillion. If there's a concurrency bug in the application which applies the algorithm the wrong account could be flagged against a legitimate image in the database. If the human involved overly trusts the system they may not validate against the actual file in the users account, or may assume the user deleted it somehow and figure its "safter" to let law enforcement figure it out. Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect. We really need to bring back comp.risks
pyuser583 5y ago
This isn’t a normal file hash. It’s not SHA or bcrypt. A wide variety of states (1s and 0s) can have the same hash. The idea is to take an image and have all of its possible derivatives create the same hash. For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash. Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa. The combinations of ones and zeros would be in the billions. If not more. And all those possible combinations of ones and zeros go back to the same “hash.” That’s extremely resourceful intensive. My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed. Or they’re using AI. They might be using AI. Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.

This is wrong - the iCloud check is against known CSAM hashes, the false positive rate is essentially zero.

hhh 5y ago

One in one trillion chance per year, per the paper on the Apple site.

fossuser 5y ago
Yeah, that’s essentially zero.
- deleted 5y ago
  [deleted]
patmcc 5y ago
I see the 'one in one trillion' but it's a bit vague; I read it as '1 in trillion chance' per image. So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.
- jeromegv 5y ago
  They have also indicated that a single match won't be enough to trigger it, so an accidental match (being that 1 in a trillion person) is not enough.
jhayward 5y ago
I would like to bet Apple's market value against that number being correct in an adversarial context.

paul_f 5y ago

This seems naive, there are always false positives

fossuser 5y ago
It’s not naive, it’s math.
- mpol 5y ago
  It's not a math problem, it's a human problem. suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.
  - fossuser 5y ago
    That isn’t CSAM. https://www.nytimes.com/interactive/2019/09/28/us/child-sex-abuse.html https://www.nytimes.com/interactive/2019/09/28/us/child-sex-... Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.
    - farmerstan 5y ago
      Who determines whether something is CSAM? How do you or I know that every single one of those images actually is CSAM? How do we know if the FBI or CIA or CCP adds hashes of innocent pics in order to pin a crime against someone?
DSingularity 5y ago
No. You can design a system where the FPR is essentially zero. Even if shitty md5 is used.
- 5y ago

2pEXgD0fZ5cF 5y ago

> the false positive rate is essentially zero I highly doubt that

DSingularity 5y ago
Based on what? Unless you point out a flaw in the math, it’s zero.
- 2pEXgD0fZ5cF 5y ago
  > Based on what? Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1] What we are left with is "trust in Apple" not "trust in math". [1]: https://news.ycombinator.com/item?id=27878333 https://news.ycombinator.com/item?id=27878333

bnj 5y ago

Yes but my understanding is that the hashes are perceptual, as opposed to cryptographic. If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate. And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.

fossuser 5y ago
I suspect that's why they have some threshold that moves the false positive rate to one in one trillion. The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.
- bnj 5y ago
  This is a great point, thanks for adding that detail.
- samatman 5y ago
  It was pretty dumb of Apple to announce both of these things at the same time. They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles. I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them. But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.
  - deleted 5y ago
    [deleted]
- lamontcg 5y ago
  It doesn't matter if the cryptographic algorithm is one in a trillion. If there's a concurrency bug in the application which applies the algorithm the wrong account could be flagged against a legitimate image in the database. If the human involved overly trusts the system they may not validate against the actual file in the users account, or may assume the user deleted it somehow and figure its "safter" to let law enforcement figure it out. Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect. We really need to bring back comp.risks

pyuser583 5y ago

This isn’t a normal file hash. It’s not SHA or bcrypt. A wide variety of states (1s and 0s) can have the same hash. The idea is to take an image and have all of its possible derivatives create the same hash. For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash. Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa. The combinations of ones and zeros would be in the billions. If not more. And all those possible combinations of ones and zeros go back to the same “hash.” That’s extremely resourceful intensive. My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed. Or they’re using AI. They might be using AI. Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.