3 ms·

In some ways I think human review is even creepier. I don’t want an algorithm looking at my private photos but I _definitely_ don’t some rando “reviewer” lookin

In some ways I think human review is even creepier. I don’t want an algorithm looking at my private photos but I _definitely_ don’t some rando “reviewer” looking at them! But I guess it all comes down to the same thing: I don’t want anyone looking at my photos unless I’ve deliberately shared my photos with them.

kemayo 5y ago
I think a lot of that does come down to levels of trust in their algorithm. Their claim is that it's staggeringly unlikely for an account to get flagged without actually containing photos which are really in the database of hashes they were provided (one in a trillion[1]). Then the only photos that the reviewers get to view are "derivatives" (they've not said what this actually means) of the photos of the photos that actually matched. Speaking for myself, if Apple is correct about those odds, I'm not personally feeling creeped out by it. If they're wrong, my opinion could change. I certainly don't have the math and security background to actually verify their claims from the white paper they posted about the system, though. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
- pseudalopex 5y ago
  Apple's claim is completely unverifiable. And people who worked with NCMEC's database said it contains non CSAM images.
  - kemayo 5y ago
    Well, okay, if you have enough non-CSAM images on your phone that are also in the NCMEC database, the reviewers will presumably get to see those specific images. They'll then go "these are obviously not naked children" and move on. If they're in the database and you also have a copy, presumably they're something like common memes that people save? That seems like it has a lower creepiness factor for reviewers to see anyway. Getting more information about the hashing function they're using would be nice. It'd make it much easier to see how actually collision-prone this is. I'd be all for them getting some external review of it published, much like the review of the security-tokens they've published. (I appreciate that it's difficult, because providing the hashing function itself to experiment with lets awful people tune their images to be just-distinct-enough that they won't match.) It's worth bearing in mind that Apple has a fairly strong motivation for the hashing to be good. They have to pay reviewers to look over these matches, and it's bad PR if it turns out that they're massively backlogged.
  - notJim 5y ago
    > And people who worked with NCMEC's database said it contains non CSAM images. Where did you see that? I tried to find more info about it, but I didn't find anything.
    - pseudalopex 5y ago
      https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
      - notJim 5y ago
        I skimmed this, and don't see anything that says the DB contains non-CSAM.
        pseudalopex 5y ago
        "The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque."
        notJim 5y ago
        Thank you. This does not support your claim, however. 1) That is an MD5 hash, not a perceptual hash. Apple is not using md5s. 2) It is a false positive, not bad info in the database. All involved acknowledge the possibility of false positives.
        pseudalopex 5y ago
        You think an MD5 collision is more likely than a wrongly classified image? NCMEC generated the hashes Apple will use by running NCMEC's collection of forbidden media through Apple's algorithm. And perceptual hashes have more collisions than cryptographic hashes. Several people said the database includes non CSAM seized in investigations.[1] [1] https://news.ycombinator.com/item?id=28069844 https://news.ycombinator.com/item?id=28069844

kemayo 5y ago

I think a lot of that does come down to levels of trust in their algorithm. Their claim is that it's staggeringly unlikely for an account to get flagged without actually containing photos which are really in the database of hashes they were provided (one in a trillion[1]). Then the only photos that the reviewers get to view are "derivatives" (they've not said what this actually means) of the photos of the photos that actually matched. Speaking for myself, if Apple is correct about those odds, I'm not personally feeling creeped out by it. If they're wrong, my opinion could change. I certainly don't have the math and security background to actually verify their claims from the white paper they posted about the system, though. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/

pseudalopex 5y ago
Apple's claim is completely unverifiable. And people who worked with NCMEC's database said it contains non CSAM images.
- kemayo 5y ago
  Well, okay, if you have enough non-CSAM images on your phone that are also in the NCMEC database, the reviewers will presumably get to see those specific images. They'll then go "these are obviously not naked children" and move on. If they're in the database and you also have a copy, presumably they're something like common memes that people save? That seems like it has a lower creepiness factor for reviewers to see anyway. Getting more information about the hashing function they're using would be nice. It'd make it much easier to see how actually collision-prone this is. I'd be all for them getting some external review of it published, much like the review of the security-tokens they've published. (I appreciate that it's difficult, because providing the hashing function itself to experiment with lets awful people tune their images to be just-distinct-enough that they won't match.) It's worth bearing in mind that Apple has a fairly strong motivation for the hashing to be good. They have to pay reviewers to look over these matches, and it's bad PR if it turns out that they're massively backlogged.
- notJim 5y ago
  > And people who worked with NCMEC's database said it contains non CSAM images. Where did you see that? I tried to find more info about it, but I didn't find anything.
  - pseudalopex 5y ago
    https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
    - notJim 5y ago
      I skimmed this, and don't see anything that says the DB contains non-CSAM.
      - pseudalopex 5y ago
        "The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque."
        notJim 5y ago
        Thank you. This does not support your claim, however. 1) That is an MD5 hash, not a perceptual hash. Apple is not using md5s. 2) It is a false positive, not bad info in the database. All involved acknowledge the possibility of false positives.
        pseudalopex 5y ago
        You think an MD5 collision is more likely than a wrongly classified image? NCMEC generated the hashes Apple will use by running NCMEC's collection of forbidden media through Apple's algorithm. And perceptual hashes have more collisions than cryptographic hashes. Several people said the database includes non CSAM seized in investigations.[1] [1] https://news.ycombinator.com/item?id=28069844 https://news.ycombinator.com/item?id=28069844