4 ms·

"This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2)" That is not bad. As a tool to filter down what apple human reviewers need to look

"This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2)" That is not bad. As a tool to filter down what apple human reviewers need to look at this is pretty good. Ultimately these images will make it to a human reviewer who can make a call as they would in any flagging system. Could a backend server side system do a more precise hash (96 bits is not a ton) prior to human review?

willis936 5y ago
2 collisions out of a million images. I'm not sure how big the CSAM database is but if it's a tens of thousands and there are millions of photos uploaded a day then Apple could have a problem on their hands. This is all extrapolating from a study that doesn't use photos representative of what people actually upload. I would suspect when most photos being uploaded are of humans the actual collision rate will be much higher.
- xadhominemx 5y ago
  They don’t take any action unless you have 30 matches in the database, which will not happen by chance.
  - willis936 5y ago
    I would agree if we assume that this very deterministic system behaved randomly.
  - AmericanChopper 5y ago
    It could happen on purpose if I intentionally send you 30 colliding images. I don’t know how iMessage handles images, but WhatsApp for example will put them directly into your photo library (and from there directly into iCloud if you’ve got syncing enabled). Perhaps I could even do that without revealing my motives to you.
    - theshrike79 5y ago
      It would STILL just mildly inconvenience the Apple wageslave tasked with checking the notifications. If you send actual CP, then that's a wholly different matter.
    - UncleMeat 5y ago
      I mean, you could send them actual illegal content too. And human reviewers are in the process. If you've got 30 matches and they are all pictures of bridges or whatever do you think the FBI is going to show up at your house?
- grlass 5y ago
  This blogpost [1] by security researcher Sarah Jamie Lewis suggests that the false positive rate could be very high: [1] https://pseudorandom.resistant.tech/obfuscated_apples.html https://pseudorandom.resistant.tech/obfuscated_apples.html
  - Grustaf 5y ago
    In their own testing they got 3 false positives out of 100 million.
ckastner 5y ago
I wouldn't be surprised if 1,431,168 photo uploads is what iCloud sees in an hour.
saithound 5y ago
Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the blog post linked from the repo [2], are both derived without making any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the practical false positive rate is now expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). A casino that makes no adversarial assumptions about the clientele could argue as follows: the probability that you accidentally win the game is much less than one in one trillion, so this game is very safe, and the House Edge is excellent [3]. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. Some of the clientele will cheat. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play [4], and the relevant probability is no longer less than 1/1000000000000. In fact, the probability that the client will win is closer to 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino. (reposted based on my comment on last week's thread [1]) [1] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [2] https://blog.roboflow.com/neuralhash-collision/ https://blog.roboflow.com/neuralhash-collision/ [3] https://wizardofodds.com/gambling/house-edge/ https://wizardofodds.com/gambling/house-edge/ [4] How to crack a linear congruential generator? http://www.reteam.org/papers/e59.pdf http://www.reteam.org/papers/e59.pdf
- SiempreViernes 5y ago
  Yeah, of course the collision rate in an adversarial dataset is likely to be much higher. But I really wonder why you think this is an important objection, do you think a lot of people want to go to the "get flagged for child porn" casino?
  - 5y ago

willis936 5y ago

2 collisions out of a million images. I'm not sure how big the CSAM database is but if it's a tens of thousands and there are millions of photos uploaded a day then Apple could have a problem on their hands. This is all extrapolating from a study that doesn't use photos representative of what people actually upload. I would suspect when most photos being uploaded are of humans the actual collision rate will be much higher.

xadhominemx 5y ago
They don’t take any action unless you have 30 matches in the database, which will not happen by chance.
- willis936 5y ago
  I would agree if we assume that this very deterministic system behaved randomly.
- AmericanChopper 5y ago
  It could happen on purpose if I intentionally send you 30 colliding images. I don’t know how iMessage handles images, but WhatsApp for example will put them directly into your photo library (and from there directly into iCloud if you’ve got syncing enabled). Perhaps I could even do that without revealing my motives to you.
  - theshrike79 5y ago
    It would STILL just mildly inconvenience the Apple wageslave tasked with checking the notifications. If you send actual CP, then that's a wholly different matter.
  - UncleMeat 5y ago
    I mean, you could send them actual illegal content too. And human reviewers are in the process. If you've got 30 matches and they are all pictures of bridges or whatever do you think the FBI is going to show up at your house?
grlass 5y ago
This blogpost [1] by security researcher Sarah Jamie Lewis suggests that the false positive rate could be very high: [1] https://pseudorandom.resistant.tech/obfuscated_apples.html https://pseudorandom.resistant.tech/obfuscated_apples.html
- Grustaf 5y ago
  In their own testing they got 3 false positives out of 100 million.

ckastner 5y ago

I wouldn't be surprised if 1,431,168 photo uploads is what iCloud sees in an hour.

saithound 5y ago

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the blog post linked from the repo [2], are both derived without making any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the practical false positive rate is now expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). A casino that makes no adversarial assumptions about the clientele could argue as follows: the probability that you accidentally win the game is much less than one in one trillion, so this game is very safe, and the House Edge is excellent [3]. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. Some of the clientele will cheat. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play [4], and the relevant probability is no longer less than 1/1000000000000. In fact, the probability that the client will win is closer to 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino. (reposted based on my comment on last week's thread [1]) [1] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [2] https://blog.roboflow.com/neuralhash-collision/ https://blog.roboflow.com/neuralhash-collision/ [3] https://wizardofodds.com/gambling/house-edge/ https://wizardofodds.com/gambling/house-edge/ [4] How to crack a linear congruential generator? http://www.reteam.org/papers/e59.pdf http://www.reteam.org/papers/e59.pdf

SiempreViernes 5y ago
Yeah, of course the collision rate in an adversarial dataset is likely to be much higher. But I really wonder why you think this is an important objection, do you think a lot of people want to go to the "get flagged for child porn" casino?
- 5y ago