4 ms·

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dw

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the blog post linked from the repo [2], are both derived without making any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the practical false positive rate is now expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). A casino that makes no adversarial assumptions about the clientele could argue as follows: the probability that you accidentally win the game is much less than one in one trillion, so this game is very safe, and the House Edge is excellent [3]. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. Some of the clientele will cheat. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play [4], and the relevant probability is no longer less than 1/1000000000000. In fact, the probability that the client will win is closer to 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino. (reposted based on my comment on last week's thread [1]) [1] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [2] https://blog.roboflow.com/neuralhash-collision/ https://blog.roboflow.com/neuralhash-collision/ [3] https://wizardofodds.com/gambling/house-edge/ https://wizardofodds.com/gambling/house-edge/ [4] How to crack a linear congruential generator? http://www.reteam.org/papers/e59.pdf http://www.reteam.org/papers/e59.pdf

SiempreViernes 5y ago
Yeah, of course the collision rate in an adversarial dataset is likely to be much higher. But I really wonder why you think this is an important objection, do you think a lot of people want to go to the "get flagged for child porn" casino?
- saithound 5y ago
  The existence of a preimage attack makes Apple's system completely useless for its nominal purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. If these new images never make it to the NCMEC database, then new CSAM content will be completely NeuralHash-proof. However, if these images eventually make their way into the NCMEC database, then everybody who has the perfectly innocent originals will be dealing with an adversarial environment.
  - SiempreViernes 5y ago
    > If these new images never make it to the NCMEC database The remark that an image might not be in the database applies equally to all CSAM pictures, and so is not actually dependent on any technical aspect of the hash at all. The fact that you can modify an image to the point that it gets a different hash isn't important compared to the (separate) issue of how you keep your database up to date. And detecting old CSAM is in any case not as important as tracking and interdicting production of new images. The second remark is good, but mainly because it reminds us to demand Apple ensures that additions are scanned for malicious images before being added to the central database. If the operator determines that a certain images collides with some known public image they can notify Apple. Apple in turn modifies the hasher to dissolve the collision, roll out a patch, and can then update the database.
  - Someone 5y ago
    > ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Even then, for that to affect _John Doe_, they would have to make 30+ images whose hash matches that of images in _John Doe’s_ iCloud account. I think that means they could target individuals, but only if they knew or could guess what photos they have in their account. They also might be able to target groups of individuals, say people who went on holiday to Paris. It would be interesting to see whether such people have enough overlap in the sets of Neuralhashes of photos they took there.
- WindyLakeReturn 5y ago
saynay 5y ago
This argument is utterly incoherent. Of course they don't include into the false positive rate images that are intentionally designed to generate positive hits, what would be the point of that? The only interesting metric is the false positive rate for normal images.
- saithound 5y ago
  It's not an argument. It's an explanation of what the number means. > The only interesting metric is the false positive rate for normal images. Wrong. The only interesting metric is the false positive rate _in practice_: i.e. how likely are false positives to affect innocents. Indeed, Apple is presenting their "one in a trillion" number as reassurance, as if it was the probability that an account that doesn't distribute CSAM gets flagged by their system. But that probability depends strongly on adversarial questions, and cannot be calculated using optimistic assumptions about all images being "normal".

SiempreViernes 5y ago

Yeah, of course the collision rate in an adversarial dataset is likely to be much higher. But I really wonder why you think this is an important objection, do you think a lot of people want to go to the "get flagged for child porn" casino?

saithound 5y ago
The existence of a preimage attack makes Apple's system completely useless for its nominal purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. If these new images never make it to the NCMEC database, then new CSAM content will be completely NeuralHash-proof. However, if these images eventually make their way into the NCMEC database, then everybody who has the perfectly innocent originals will be dealing with an adversarial environment.
- SiempreViernes 5y ago
  > If these new images never make it to the NCMEC database The remark that an image might not be in the database applies equally to all CSAM pictures, and so is not actually dependent on any technical aspect of the hash at all. The fact that you can modify an image to the point that it gets a different hash isn't important compared to the (separate) issue of how you keep your database up to date. And detecting old CSAM is in any case not as important as tracking and interdicting production of new images. The second remark is good, but mainly because it reminds us to demand Apple ensures that additions are scanned for malicious images before being added to the central database. If the operator determines that a certain images collides with some known public image they can notify Apple. Apple in turn modifies the hasher to dissolve the collision, roll out a patch, and can then update the database.
- Someone 5y ago
  > ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Even then, for that to affect _John Doe_, they would have to make 30+ images whose hash matches that of images in _John Doe’s_ iCloud account. I think that means they could target individuals, but only if they knew or could guess what photos they have in their account. They also might be able to target groups of individuals, say people who went on holiday to Paris. It would be interesting to see whether such people have enough overlap in the sets of Neuralhashes of photos they took there.
WindyLakeReturn 5y ago

saynay 5y ago

This argument is utterly incoherent. Of course they don't include into the false positive rate images that are intentionally designed to generate positive hits, what would be the point of that? The only interesting metric is the false positive rate for normal images.

saithound 5y ago
It's not an argument. It's an explanation of what the number means. > The only interesting metric is the false positive rate for normal images. Wrong. The only interesting metric is the false positive rate _in practice_: i.e. how likely are false positives to affect innocents. Indeed, Apple is presenting their "one in a trillion" number as reassurance, as if it was the probability that an account that doesn't distribute CSAM gets flagged by their system. But that probability depends strongly on adversarial questions, and cannot be calculated using optimistic assumptions about all images being "normal".