4 ms·

It's worth bearing in mind that the human review step does mean that a government can't just slip stuff in without securing Apple's cooperation (including train

by kemayo 5y ago

It's worth bearing in mind that the human review step does mean that a government can't just slip stuff in without securing Apple's cooperation (including training their review staff about all the political content they have to look for). Otherwise the reviewers would presumably just go "huh, that's weird, this Winnie the Pooh meme definitely isn't child porn" and move on. Can a government secure Apple's cooperation in that? I have no idea. But it does make a useful subversion of the hash database a more complicated thing to accomplish.

elliekelly 5y ago
In some ways I think human review is even creepier. I don’t want an algorithm looking at my private photos but I _definitely_ don’t some rando “reviewer” looking at them! But I guess it all comes down to the same thing: I don’t want anyone looking at my photos unless I’ve deliberately shared my photos with them.
- kemayo 5y ago
  I think a lot of that does come down to levels of trust in their algorithm. Their claim is that it's staggeringly unlikely for an account to get flagged without actually containing photos which are really in the database of hashes they were provided (one in a trillion[1]). Then the only photos that the reviewers get to view are "derivatives" (they've not said what this actually means) of the photos of the photos that actually matched. Speaking for myself, if Apple is correct about those odds, I'm not personally feeling creeped out by it. If they're wrong, my opinion could change. I certainly don't have the math and security background to actually verify their claims from the white paper they posted about the system, though. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
  - pseudalopex 5y ago
    Apple's claim is completely unverifiable. And people who worked with NCMEC's database said it contains non CSAM images.
    - kemayo 5y ago
      Well, okay, if you have enough non-CSAM images on your phone that are also in the NCMEC database, the reviewers will presumably get to see those specific images. They'll then go "these are obviously not naked children" and move on. If they're in the database and you also have a copy, presumably they're something like common memes that people save? That seems like it has a lower creepiness factor for reviewers to see anyway. Getting more information about the hashing function they're using would be nice. It'd make it much easier to see how actually collision-prone this is. I'd be all for them getting some external review of it published, much like the review of the security-tokens they've published. (I appreciate that it's difficult, because providing the hashing function itself to experiment with lets awful people tune their images to be just-distinct-enough that they won't match.) It's worth bearing in mind that Apple has a fairly strong motivation for the hashing to be good. They have to pay reviewers to look over these matches, and it's bad PR if it turns out that they're massively backlogged.
    - notJim 5y ago
      > And people who worked with NCMEC's database said it contains non CSAM images. Where did you see that? I tried to find more info about it, but I didn't find anything.
      - pseudalopex 5y ago
        https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
        notJim 5y ago
        I skimmed this, and don't see anything that says the DB contains non-CSAM.
        pseudalopex 5y ago
        "The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque."
makeitdouble 5y ago
It’s been a while it hasn’t been discussed, but isn’t there laws to give gov special agencies the power to force private entities to cooperate, with potentially a ban on officially recognizing the request ever happened in the first place ? And if memory serves well it doesn’t need to be Apple as a whole cooperating, a single employee with oversee power could be enough.
zionic 5y ago
Due to existing laws, Apple will be forced to make their human reviewers be active LEOs. Their approval/report rate will make a FISA court judge blush.

by kemayo 5y ago

elliekelly 5y ago

In some ways I think human review is even creepier. I don’t want an algorithm looking at my private photos but I _definitely_ don’t some rando “reviewer” looking at them! But I guess it all comes down to the same thing: I don’t want anyone looking at my photos unless I’ve deliberately shared my photos with them.

kemayo 5y ago
I think a lot of that does come down to levels of trust in their algorithm. Their claim is that it's staggeringly unlikely for an account to get flagged without actually containing photos which are really in the database of hashes they were provided (one in a trillion[1]). Then the only photos that the reviewers get to view are "derivatives" (they've not said what this actually means) of the photos of the photos that actually matched. Speaking for myself, if Apple is correct about those odds, I'm not personally feeling creeped out by it. If they're wrong, my opinion could change. I certainly don't have the math and security background to actually verify their claims from the white paper they posted about the system, though. [1]: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
- pseudalopex 5y ago
  Apple's claim is completely unverifiable. And people who worked with NCMEC's database said it contains non CSAM images.
  - kemayo 5y ago
    Well, okay, if you have enough non-CSAM images on your phone that are also in the NCMEC database, the reviewers will presumably get to see those specific images. They'll then go "these are obviously not naked children" and move on. If they're in the database and you also have a copy, presumably they're something like common memes that people save? That seems like it has a lower creepiness factor for reviewers to see anyway. Getting more information about the hashing function they're using would be nice. It'd make it much easier to see how actually collision-prone this is. I'd be all for them getting some external review of it published, much like the review of the security-tokens they've published. (I appreciate that it's difficult, because providing the hashing function itself to experiment with lets awful people tune their images to be just-distinct-enough that they won't match.) It's worth bearing in mind that Apple has a fairly strong motivation for the hashing to be good. They have to pay reviewers to look over these matches, and it's bad PR if it turns out that they're massively backlogged.
  - notJim 5y ago
    > And people who worked with NCMEC's database said it contains non CSAM images. Where did you see that? I tried to find more info about it, but I didn't find anything.
    - pseudalopex 5y ago
      https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html https://www.hackerfactor.com/blog/index.php?/archives/929-On...
      - notJim 5y ago
        I skimmed this, and don't see anything that says the DB contains non-CSAM.
        pseudalopex 5y ago
        "The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque."

makeitdouble 5y ago

It’s been a while it hasn’t been discussed, but isn’t there laws to give gov special agencies the power to force private entities to cooperate, with potentially a ban on officially recognizing the request ever happened in the first place ? And if memory serves well it doesn’t need to be Apple as a whole cooperating, a single employee with oversee power could be enough.

zionic 5y ago

Due to existing laws, Apple will be forced to make their human reviewers be active LEOs. Their approval/report rate will make a FISA court judge blush.