28 ms·

ImageNet contains naturally occurring Apple NeuralHash collisions

860 points by yeldarb 5y ago

MilaM 5y ago
I have a suggestion for Apple. Maybe they could use this technique in a way that benefits all their customers by finally solving the problem of duplicate image imports in Photos.app. I have a couple of hundred duplicate images in my Library because of problems during local import from my iPhone into Photos.
- CharlesW 5y ago
  PhotoSweeper[1] is great for dealing with this. (Not affiliated, just a happy customer trying to manage 100K photos.) [1] https://overmacs.com/ https://overmacs.com/
deleted 5y ago
[deleted]
zug_zug 5y ago
Here's one concern -- Neural Hash is finding false positives with images that look very similar. What if an underage girl takes a selfie (or over-age girl) and the pose or background features are similar enough to trigger a false collision. Then Apple is going to a manual step where they are able to look at a low-res version of somebody's private photos as I understand it. In my opinion that last step is not okay. I suppose the 30-image threshhold is a mitigating factor, really imo Apple is making their problem into my problem. I want to purchase from a company that offers me the peace of mind not to even have to think about such concerns, price isn't an obstacle. If Apple can't cater to my needs I hope another company will.
- CharlesW 5y ago
  > I want to purchase from a company that offers me the peace of mind not to even have to think about such concerns, price isn't an obstacle. Since this only affects people using iCloud Photos, have you considered not doing that, or using an alternative that you believe isn't doing this?
alfalfasprout 5y ago
There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).
- criticaltinker 5y ago
  The OP mentions that two countries have to agree to add a file to the list, but your concern is definitely valid: > Perhaps the most concerning part of the whole scheme is the database itself. Since the original images are (understandably) not available for inspection, it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse.
  - ulzeraj 5y ago
    What if those two countries can be Poland and Hungary? These two countries have been passing lots of laws to ostracize and criminalize pro-LGBT content and are friendly to each other.
    - tehnub 5y ago
      Apple did mention in their security thread model document [0] this: Apple will also refuse all requests to instruct human reviewers to file reports for anything other than CSAM materials for accounts that exceed the match threshold. [0]: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
      - nybble41 5y ago
        That's too little, too late. By the time any human reviewer can evaluate whether the images should have been in the database the encryption on the backups has already been circumvented and other parties have already been given access to your private files.
criticaltinker 5y ago
> In order to test things, I decided to search the publicly available ImageNet dataset for collisions between semantically different images. I generated NeuralHashes for all 1.43 million images and searched for organic collisions. By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. > This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. It's great to see the ingenuity and attention this whole debacle is receiving from the community. Maybe it will lead to advances in perceptual hashing (and also advances in consumer awareness of tech related privacy issues).
- bawolff 5y ago
  ImageNet is a very well-known data set. Are we sure apple didn't test on it when designing this algorithm?
  - slg 5y ago
    >This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. Apple reported a pretty similar collision rate so maybe they did.
    - yeldarb 5y ago
      Their reported collision rate was against their CSAM hash database[1]. > In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives. [1] https://tidbits.com/2021/08/13/new-csam-detection-details-emerge-following-craig-federighi-interview/ https://tidbits.com/2021/08/13/new-csam-detection-details-em...
matsemann 5y ago
> By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2 Someone got more details on that? How does the birthday paradox come into play here?
- deleted 5y ago
  [deleted]
- throwaway287391 5y ago
  Yeah I'm unclear on why this even requires a clever algorithm. I'd think that given the 1.4 million precomputed hashes, a simple/naive Python function (<10 lines) that builds a dict mapping hash -> (list of images with that hash) could surface all the collisions in a few seconds. (It's a cool article though! I'm glad someone tested this.) Edit: I'm procrastinating so I tried it. It's 8 lines including IO/parsing and runs in 2.8 seconds on my laptop: import collections hash_to_filenames = collections.defaultdict(list) with open('hashes.txt') as f: for line in f.readlines(): filename, hash = line.strip().split() hash_to_filenames[hash].append(filename) dupes = {h: fs for h, fs in hash_to_filenames.items() if len(fs) > 1} print(f'Done. Found {len(dupes)} dupes.') (hashes.txt is from the zip in the github repo, and it finds 8865 dupes which looks almost right from the article text (8272 + 595 = 8867).)
- aaaaaaaaaaab 5y ago
  https://en.m.wikipedia.org/wiki/Birthday_attack https://en.m.wikipedia.org/wiki/Birthday_attack
- sweezyjeezy 5y ago
  He means that even though the chance of two random images colliding is ~ 1/ 2 trillion, once you get up to a set of order sqrt(2 trillion) you have a good chance of having a collision amongst all pairs.
  - throwaway287391 5y ago
    That explains the "birthday paradox" part, what I'm unclear on is the need for a "collision search algorithm" that isn't just "build a hashmap" which should take roughly O(N) time. (I suppose it could just be that, but I'm surprised it's even mentioned in that case. In my uncle(?) comment I wrote an 8 line Python implementation that runs in 3 seconds on my laptop.)
JimBlackwood 5y ago
It is actually really cool to see that those small changes generate the same hash. At least those parts are working well.
endisneigh 5y ago
It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’ve done something bad without evidence and imprison you. It happens all the time. It’s even happened in America. Just look up DNA exonerations - people have had DNA on the scene that literally proves their innocence and they’re still locked up. People should care about their governments being corrupt, not necessarily about this.
- Dah00n 5y ago
  Lots of things start out like this, then later they start looking for or blocking other things. Pirated movies, files leaked from three letter agencies, a picture of the president with a ** in his mouth, etc. Even if this is 100% bullet-proof it is still enough that Apple should be seen as privacy invading and leaking everything to the government (and others later on) as it will be abused if implemented everywhere. This isn't irrelevant because of other things happening that are also bad. This is added on-top of those broken system, like the ones in the US you mention.
- minitoar 5y ago
  The damage is already done by the time it gets to the point of devices being confiscated.
  - endisneigh 5y ago
    Can you point to a single instance of that happening where it was due to a false positive?
    - bawolff 5y ago
      In a proposed system that has yet to be deployed?
aaaaaaaaaaab 5y ago
>By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2 n(log n)? I thought the birtday paradox lets you expect collisions after sqrt(n) tries…
- yeldarb 5y ago
  Different things; birthday paradox was testing them against each other, turning it into a sorting problem was the other.
jeffbee 5y ago
Following the topic because it's interesting, but I'm bothered by the chain of reasoning necessary to reach these conclusions. How sure are we that the means of dumping the model from iOS and converting it to ONNX results in a faithful representation of what NeuralHash actually does?
- skygazer 5y ago
  FWIW, Apple says that’s not even the NN hash function they’ll be using. This one has been buried in iOS since 2019, so it may have been a prototype. Although dumping the NN image seems straightforward, so this seems a faithful copy of that older hash function. (There’s been a suggestion that it’s sensitive to floating point processor implementations, so small hash differences may occur between different CPU architectures.)
- tyingq 5y ago
  I agree in theory, but the burden of proof shouldn't be on outsiders that have no other choice than to extrapolate. Proposing something like this running on an end-user's personally owned device should have a really high bar.
stickfigure 5y ago
> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Is anyone so naive that they believe the CCP wouldn't deliver such an ultimatum? Apple's position seems to completely ignore recent Chinese history.
- tyingq 5y ago
  >(like a foreign government) It seems like it wouldn't take that. If you can generate a colliding pair of images, you could probably create a pair where one of the images might get attention with child porn groups and thus, shared around enough to end up in the CSAM database. And where the other was innocuous.
- bsdetector 5y ago
  > If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Or maybe China already said "put in this CSAM check or you can't make or sell phones in China". Since Apple's position is contrary to their previous privacy policy and doesn't seem to make a lot of sense, it's quite possible extortion already happened (and not necessarily by China).
  - ballenf 5y ago
    I'd honestly believe such pressure came from US domestic intelligence or law enforcement agencies just as easily as from China.
    - adventured 5y ago
      It wouldn't specifically be from domestic intelligence, it would be from a powerful member of Congress with a relationship to Apple (specifically the board/management), acting as a go-between that would try to politically lay out the situation for them. Hey Apple, we can either turn up the anti-trust heat by a lot, or we can turn it down, which is it going to be? Except it would be couched in intellectually dishonest language meant to preserve a veneer that the US Government isn't a violent, quasi-psychotic bully ready to bash your face in if you don't do what they're asking. The interactions with intelligence about the new program would begin after they acquiesce to going along. It's too easy. There's an extraordinary amount of wilful naivety in the US about the nature of the government and its frequent power abuses (what it's willing to do), despite the rather comically massive demonstration of said abuses spanning the entire post WW2 era. Every time it happens the wilfully naive crowd feigns surprise.
pizzaknife 5y ago
the world of forensics is something im not familiar with, a couple questions: - When a hash matches (correctly or incorrectly), how is said image reported? is the matching image passed on to a human to verify or? - what is the survey size / who(m) are subject to this CSAM net?
- rootusrootus 5y ago
  after ~30 matches, only the matching images are passed on to a human for visual verification. only images uploaded to iCloud are subject to matching
  - Dah00n 5y ago
    If they pass CSAM verfied by hash on to human verification inside Apple they break the law. Not even the FBI are allowed to do that. Only NMCEC is an allowed recipient by US federal law.
    - nonbirithm 5y ago
      > Not even the FBI are allowed to do that. The FBI was given clearance to take over a website serving CSAM in order to catch more users of the site. As such, the FBI has technically distributed CSAM in the past. https://www.dallasnews.com/news/crime/2017/01/17/the-fbi-ran-a-child-porn-site-to-catch-predators-and-now-the-accused-are-crying-foul/ https://www.dallasnews.com/news/crime/2017/01/17/the-fbi-ran...
    - rootusrootus 5y ago
      Seems to be a misunderstanding between what the law appears to say and what the actual practice is. Law enforcement's interest is not served by trying to prosecute moderators or companies acting in good faith because they have CSAM in their possession.
      - nybble41 5y ago
        There is a difference between moderators manually identifying illegal content in a stream of mostly-legal material and a process where content which has already been matched against the database and classified as almost-certainly-illegal is subjected to further internal review.
rm999 5y ago
tldr: this is expected. The article addresses everything I'm about to say, but I think the lede is buried: >Apple's NeuralHash perceptual hash function performs its job better than I expected When you are just looking for any two collisions in 100M x 100M comparisons, of course you'll find a small number of positives, Apple said as much. The number of expected collisions will scale linearly with the number of 'bad' images, which is not 100M. Assuming it's 100k, we'd expect 1000x fewer collisions in ImageNet, or ~0.002 collisions, which is effectively 0. It's the artificial images that will potentially sink all this, not a very low rate of naturally occurring hash collisions.
biesnecker 5y ago
> This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. The number of images in that database could well be far in excess of a million. According to NCMEC [1], in 2020 65.4 million files that were reported to them, and "[s]ince the program inception in 2002, CVIP [child victim identification project] has reviewed more than 330 million images and videos." Of course many of those were duplicate but it would be entirely unsurprised if there were more than a million original files. [1] https://www.missingkids.org/ourwork/impact https://www.missingkids.org/ourwork/impact
tehnub 5y ago
> In order to test things, I decided to search the publicly available ImageNet dataset for collisions between semantically different images. I generated NeuralHashes for all 1.43 million images and searched for organic collisions. By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. I don't know what the author means by "taking advantage of the birthday paradox". If they're referring to the "birthday attack" [0], I don't think it makes sense. The birthday attack is a strategy that helps you find a collision without hashing every single image, but he states that he already generated NeuralHashes for all 1.43 million images. Furthermore, isn't there a simple linear time algorithm to detect collisions given that you already have all the hashes? Iterate over your NeuralHashes and put them into a hash table where the NeuralHash is the key, and the number of occurrences is the value. Whenever you insert something into a bucket and there's already something there, you have a neural hash collision. [0]: https://en.wikipedia.org/wiki/Birthday_problem#Probability_table https://en.wikipedia.org/wiki/Birthday_problem#Probability_t...
- tibbar 5y ago
  1) the point of the birthday paradox is that even if two elements of a set are unlikely to overlap, in a large set it is much more likely than perhaps intuitive that some pair of elements overlap. 2) I’m assuming he did something like putting all the hashes in a list and sorting them, which is at least better than looking at each pair. As you say, it’s not optimal and also doesn’t seem particularly worth including in the otherwise very interesting post
- sweezyjeezy 5y ago
  By "taking advantage of the Birthday Paradox" - he means that even though the chance of two random images colliding is ~ 1/1 trillion, if you have a set of size ~ sqrt(1 trillion) you have a good chance of having a collision amongst all pairs.
  - tehnub 5y ago
    But did the birthday paradox inform his procedure in any way? I guess he wouldn't have even attempted to do this at all if he didn't think it was likely to find collisions.
tbenst 5y ago
I’m much more worried about adversarial attacks rather than accidental false positives. For example, a year ago I was trying to track down how some unscrupulous photos ended up in my photo library. I finally realized that Telegram had, without my knowing permission, been adding all photos I had received in a crypto discussion group to my Apple photo library. There seem many ripe opportunities for targeting.
- swiley 5y ago
  This kind of thing is why you really can’t have conversations on phones in confidence. You really have no idea what the chat client on your phone (much less the other person’s) is doing.
nanis 5y ago
It sounds like there is code that enables anyone to compute the perceptual hash for an image. Is this code published somewhere?
- yeldarb 5y ago
  The instructions are here: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX This was discussed on HN yesterday here (and a few other front-page stories): https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391
  - nanis 5y ago
    Got it ... Here's the part I was missing: > You will need 4 files from a recent macOS or iOS build: neuralhash_128x96_seed1.dat NeuralHashv3b-current.espresso.net NeuralHashv3b-current.espresso.shape NeuralHashv3b-current.espresso.weights So, for clarity, Apple did not publish the model/code for others to be able to use/test etc. Someone found a clever way to convert the stored model into an open format and therefore others with access to those files are able to experiment.
smithza 5y ago
This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. If the author was comparing 2 trillion pictures of people, or children specifically, I think this false-positive rate would be different and arguably much higher. The reasons are obvious: humans are similar in dimensions to eachother and are much more likely to match in the same way the hatchet and nematode matched. I do not presume such a finding of photos is easy to come by but I wish the author put details on the sample set.
- yeldarb 5y ago
  The sample set is ImageNet, which is a well-known dataset in Computer Vision and is available for download here: https://www.kaggle.com/c/imagenet-object-localization-challenge https://www.kaggle.com/c/imagenet-object-localization-challe... I'd love to see this work extended; if you find additional collisions in the wild please submit a PR to the repo (please do not submit artificially generated adversarial images): https://github.com/roboflow-ai/neuralhash-collisions https://github.com/roboflow-ai/neuralhash-collisions For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes) [1] https://tidbits.com/2021/08/13/new-csam-detection-details-emerge-following-craig-federighi-interview/ https://tidbits.com/2021/08/13/new-csam-detection-details-em... > In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives.
  - smithza 5y ago
    For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes) This is an important note. Is it the case that this algorithm is trained for humans or not? the 1/trillion false-positive rate might imply it is trained with a broader set. Thank you for those helpful tidbits.
28194608 5y ago
You guys are helping Apple not to fuckup their implementation by finding bugs. It would be great if these collision posts happened after they fully rolled out the update.
- foepys 5y ago
  Once the code landed in iOS and is activated, it is too late.
hyperbovine 5y ago
Err, most of these are not naturally occurring pairs since in each case the images differ by a human manipulation (resolution reduction, drawing an arrow, changing the watermark, changing the aspect ratio)---which I'm guessing is viewed as a feature not a bug by the designers of this system. The axe and the nematode comes closest, and even then, they are low-res and visually similar. What would be far more concerning is a hash collision between a picture of a piano and a picture of an elephant, but nothing like that is happening here.
vhold 5y ago
> By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. I think you could just do "sort | uniq -c | sort -nr" on the neuralhash values to find the most frequently occurring ones pretty fast?
- occamrazor 5y ago
  That’s exactly the O(n log n) algorithm.
kizer 5y ago
Rehash everything using 512 bits or something.
- mrtranscendence 5y ago
  Would that make a difference? Technically 96 bits is more than enough to assign a unique hash to every image that ever existed, many times over.
oconnore 5y ago
If it's so easy to modify NeuralHashes, won't "CSAM networks" just rotate the hashes of their "collections"? If you can make an innocent picture collide with a CSAM picture, presumably you can also edit a CSAM picture to have a random hash not in the database?
- mrtranscendence 5y ago
  Through chatting and reading threads I get the impression that many folks think that pedophiles are all smart, extremely tech savvy, operating in networks that share information and collaborate to fool law enforcement. There may be some of that, but there are also plenty of Joe Schmoes who download or trade CSAM material without being especially clever about it. I remember when I was a teenager downloading porn I found more than a few probably-illegal photos just by accident*. If you want to find such images you can, it doesn't take an organized "network". * edit - this was 25 years ago, admittedly, and much of what I downloaded came from randos on AOL. Things might be different now.
tgsovlerkhgsel 5y ago
> Apple now has over 1.5 billion users so we are talking about a large pool of users at stake which increases the likelihood of even a low probability event manifesting This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. What's even worse, when those individuals are facing whatever they're facing, they'll have to argue against the one-in-a-billion odds. If there are jurisdictions where defense lawyers don't get access to the images their client is accused of, and prosecutors and judges don't look at the images but only at the surrounding evidence (which says this person is guilty, with 99.9999999% certainty), Apple may have built a system that's statistically expected to send an innocent person to prison.
- mrtranscendence 5y ago
  > This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually CSAM then it won't ever make it to law enforcement.
  - tgsovlerkhgsel 5y ago
    What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be flagged), confusion, or malice? 1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system. If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc. On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.
dan-robertson 5y ago
It didn’t seem to take long for the weights for Apple’s network to be discovered. And I suppose they must send the banned hashes to the client for checking too. So I expect that list will be discovered and published soon too (unless they have some way to keep them secret?) I think one important question is: how reversible is Apple’s perceptual hash? For example, my understanding of Microsoft’s PhotoDNA is that their perceptual hash has been reverse-engineered and that one could go backwards from a hash to a blurry image. But also it is very hard to get the list of PhotoDNA hashes for the NCMEC database. In other words, are Apple unintentionally releasing enough information to reconstruct a bunch of blurry CSAM?
- yeldarb 5y ago
  It doesn't seem like it; check out the adversarially constructed images here. They don't look anything like the original despite perfectly matching the NeuralHash: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
  - dan-robertson 5y ago
    Right obviously the hashing isn’t going to be injective and therefore there are lots of silly images that hash to a given value. The question is whether it is possible to efficiently find plausible images with a given hash. Think more like a “deep dream” than these adversarial attacks.
- fortenforge 5y ago
  The hashes sent to the client are cryptographically blinded, making it impossible for the client to determine the original CSAM hashes.
  - dan-robertson 5y ago
    This is the information I was missing, thanks.
- kccqzy 5y ago
saithound 5y ago
Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). In an ordinary, non-adversarial context, the probability that you win the game is much less than one in one trillion, so this game is very safe for the casino. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play, and the relevant probability is no longer one in one trillion. The relevant number is 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.
- mrtranscendence 5y ago
  Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.
  - e40 5y ago
    > The end result is that some peon at Apple has to look at the images and mark them as not CSAM Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.
    - 5y ago
jb1991 5y ago
> Apple's NeuralHash perceptual hash function performs its job better than I expected and the false-positive rate on pairs of ImageNet images is plausibly similar to what Apple found between their 100M test images and the unknown number of NCMEC CSAM hashes.
Grakel 5y ago
Maybe this is a shot in the arm for anonymous devices. Somebody buy a bunch of GSM devices and resell for cash.
ctur 5y ago
The more interesting point about hash collisions is probably less about accidental clashes and more about intentional clashes. If the hashes in the CSAM database were known publicly, and people began generating intentional clashes with innocuous images, and those images were on many phones, it could basically DOS whatever manual process Apple creates. Basically it could become an arms race where, say, free speech advocates convince people to just keep and share some images and overwhelm the process. Then Apple adapts to new hashes, blocklists known false positives, and the cycle repeats.
joelbondurant 5y ago
Apple employees will do anything to keep their jobs looking nat kiddie porn.
deleted 5y ago
[deleted]
tartoran 5y ago
I am not exactly sure if this is how this works but it appears to me that all the hashes of your photos get uploaded to a server and was wondering if it is possible to reverse the hashes to be able to deduce what's in each photo hashed..
- ilogik 5y ago
  no and no
  - tartoran 5y ago
    Your answer without an explanation doesn't amount to anything more than a scant opinion. "no and no" isn't worth commenting on hackernews in my opinion. As I stated above, I am not very familiar with how this bleeding edge technology works but am not learning anything new from your response. If you wanted to expand I'd appreciate it
cheald 5y ago
The demonstrated ability to produce collisions at will should be an instant show-stopper for this feature. If a bad actor can send targets innocent images which hash to CSAM, you essentially have an upgraded secret swatting mechanism that people will use to abuse others.
kebman 5y ago
Oh boy... I'm just here from watching WEF videos on YouTube. I'm gonna go hide in a cave now.
solidgol 5y ago
I think the headline is misleading. The point of the article is that the algorithm that Apple is using works as well as Apple says it does, and the headline implies that it doesn't.
- dang 5y ago
  What would be a better (i.e. more accurate and neutral) title?
  - solidgol 5y ago
    A reply by the moderator! I am honored. I think the headline should be: ImageNet contains about as many naturally occurring NeuralHash collisions as Apple predicts
    - yeldarb 5y ago
      Hey, OP here - while that is also true, I think the average observer would think this number _should_ be zero. The fact that any exist at all is surprising (but the fact that only two exist is also surprising; I expected to find either zero or lots when I started the script).
orange_puff 5y ago
What we know from this well written and helpful article; The false positive rate Apple told us their algorithm had seems to be accurate. If a machine learning model is extracted from the OS it exists on, it will be much easier to generate adversarial attacks. For example, a neural net's cost function is just a multivariate function with weights as its input. To figure out how to move those weights (positively or negatively), the gradient of the function is calculated and the weights are nudged in the opposite direction (gradient is the direction of the largest growth of a function, we are trying to minimize the cost). Now, assume we are given a cost function and the weights are constant, now, the input can be the image. So, we take the gradient of the cost function with respect to the image pixels and can now see how we should nudge those to maximize the cost. Apple will absolutely need to protect against adversarial attacks for this to be viable. I'm hopeful.
tentacleuno 5y ago
I'm really starting to feel out of touch with Hacker News. Apple seems to be consistently on the front page somehow. Surely other people are sick of hearing about Apple too? They do say any press is good press, though.
GatorD42 5y ago
Wouldn’t the most likely attack be an angry (ex) spouse with physical access to a target’s phone placing images on it (and uploading to iCloud)? I haven’t seen this seemingly likely scenario discussed much.
- judge2020 5y ago
  two scenarios: - Those images would either have to be actual CSAM (illegal to distribute in the first place and generally hard to get) - fake collision images are placed instead, and all that would do is push the multiple fake CSAM images to Apple's human reviewers who would then likely mark it as a false positive/spam. If the Apple reviewer makes a mistake and does report it as CSAM them the only risk (in the US) is law enforcement having probably cause for search & seizure to look for other CSAM.
  - GatorD42 5y ago
    The first scenario seems much more plausible and genuinely worrying.
joemountain 5y ago
From the article, in the conclusion section: "Apple's NeuralHash perceptual hash function performs its job better than I expected" Just want to surface that because the title seems a little sensational based on the conclusion.
IlliOnato 5y ago
> Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. They mean Russia and Belarus would need to agree to add a file to the list? Yeah, this is a very hard barrier to overcome! /s
NoPicklez 5y ago
As a completely unrelated side note separate to Apple's implementation. I just read today that OnlyFans also contribute hashes to the same organisation, I wonder how many other companies have been doing the same for years. Haven't see any uproar about this elsewhere for other services. For example, CloudFare has had this as an option on their environment for years.
fluidcruft 5y ago
I guess I'd care if any of the collisions weren't of things that I can vaguely see as looking similar. It's not like a nail or a ski looks like child porn. I'm actually impressed. Who cares that they collide. The manufactured collisions are obviously a problem though if I understand correctly that it means CP could be hidden by making it collide with well-known (and likely whitelisted) images.
Threeve303 5y ago
I’m honestly fine with Apple doing this if it’s for the stated reasons. But experience has shown me that this is essentially copyright infringement enforcement dressed up in some emotional blackmail. Of course the point where this became inevitable was when Apple began creating and distributing media.

860 points by yeldarb 5y ago

I have a suggestion for Apple. Maybe they could use this technique in a way that benefits all their customers by finally solving the problem of duplicate image imports in Photos.app. I have a couple of hundred duplicate images in my Library because of problems during local import from my iPhone into Photos.

CharlesW 5y ago
PhotoSweeper[1] is great for dealing with this. (Not affiliated, just a happy customer trying to manage 100K photos.) [1] https://overmacs.com/ https://overmacs.com/

deleted 5y ago

[deleted]

zug_zug 5y ago

Here's one concern -- Neural Hash is finding false positives with images that look very similar. What if an underage girl takes a selfie (or over-age girl) and the pose or background features are similar enough to trigger a false collision. Then Apple is going to a manual step where they are able to look at a low-res version of somebody's private photos as I understand it. In my opinion that last step is not okay. I suppose the 30-image threshhold is a mitigating factor, really imo Apple is making their problem into my problem. I want to purchase from a company that offers me the peace of mind not to even have to think about such concerns, price isn't an obstacle. If Apple can't cater to my needs I hope another company will.

CharlesW 5y ago
> I want to purchase from a company that offers me the peace of mind not to even have to think about such concerns, price isn't an obstacle. Since this only affects people using iCloud Photos, have you considered not doing that, or using an alternative that you believe isn't doing this?

alfalfasprout 5y ago

There's been a lot of focus on the likelihood of collisions and whether someone could upload eg; an image with a matching hash to your device to "set you up", etc. But what's still extremely concerning is that there is still no guarantee that the hash list used can't be coopted for another purpose (eg; politically insensitive content).

criticaltinker 5y ago
The OP mentions that two countries have to agree to add a file to the list, but your concern is definitely valid: > Perhaps the most concerning part of the whole scheme is the database itself. Since the original images are (understandably) not available for inspection, it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse.
- ulzeraj 5y ago
  What if those two countries can be Poland and Hungary? These two countries have been passing lots of laws to ostracize and criminalize pro-LGBT content and are friendly to each other.
  - tehnub 5y ago
    Apple did mention in their security thread model document [0] this: Apple will also refuse all requests to instruct human reviewers to file reports for anything other than CSAM materials for accounts that exceed the match threshold. [0]: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
    - nybble41 5y ago
      That's too little, too late. By the time any human reviewer can evaluate whether the images should have been in the database the encryption on the backups has already been circumvented and other parties have already been given access to your private files.

criticaltinker 5y ago

> In order to test things, I decided to search the publicly available ImageNet dataset for collisions between semantically different images. I generated NeuralHashes for all 1.43 million images and searched for organic collisions. By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. > This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. It's great to see the ingenuity and attention this whole debacle is receiving from the community. Maybe it will lead to advances in perceptual hashing (and also advances in consumer awareness of tech related privacy issues).

bawolff 5y ago
ImageNet is a very well-known data set. Are we sure apple didn't test on it when designing this algorithm?
- slg 5y ago
  >This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. Apple reported a pretty similar collision rate so maybe they did.
  - yeldarb 5y ago
    Their reported collision rate was against their CSAM hash database[1]. > In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives. [1] https://tidbits.com/2021/08/13/new-csam-detection-details-emerge-following-craig-federighi-interview/ https://tidbits.com/2021/08/13/new-csam-detection-details-em...

matsemann 5y ago

> By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2 Someone got more details on that? How does the birthday paradox come into play here?

deleted 5y ago
[deleted]
throwaway287391 5y ago
Yeah I'm unclear on why this even requires a clever algorithm. I'd think that given the 1.4 million precomputed hashes, a simple/naive Python function (<10 lines) that builds a dict mapping hash -> (list of images with that hash) could surface all the collisions in a few seconds. (It's a cool article though! I'm glad someone tested this.) Edit: I'm procrastinating so I tried it. It's 8 lines including IO/parsing and runs in 2.8 seconds on my laptop: import collections hash_to_filenames = collections.defaultdict(list) with open('hashes.txt') as f: for line in f.readlines(): filename, hash = line.strip().split() hash_to_filenames[hash].append(filename) dupes = {h: fs for h, fs in hash_to_filenames.items() if len(fs) > 1} print(f'Done. Found {len(dupes)} dupes.') (hashes.txt is from the zip in the github repo, and it finds 8865 dupes which looks almost right from the article text (8272 + 595 = 8867).)
aaaaaaaaaaab 5y ago
https://en.m.wikipedia.org/wiki/Birthday_attack https://en.m.wikipedia.org/wiki/Birthday_attack
sweezyjeezy 5y ago
He means that even though the chance of two random images colliding is ~ 1/ 2 trillion, once you get up to a set of order sqrt(2 trillion) you have a good chance of having a collision amongst all pairs.
- throwaway287391 5y ago
  That explains the "birthday paradox" part, what I'm unclear on is the need for a "collision search algorithm" that isn't just "build a hashmap" which should take roughly O(N) time. (I suppose it could just be that, but I'm surprised it's even mentioned in that case. In my uncle(?) comment I wrote an 8 line Python implementation that runs in 3 seconds on my laptop.)

JimBlackwood 5y ago

It is actually really cool to see that those small changes generate the same hash. At least those parts are working well.

endisneigh 5y ago

It doesn’t matter if there are collisions if the two images don’t actually look the same. Do people honestly believe a single CSAM flag from an “innocent” image is going to result in someone going to prison in America? PhotoDNA has existed for over a decade doing the same thing with no instances that I have heard of. If some corrupt government wants to get you they don’t need this. They can just unilaterally say you’ve done something bad without evidence and imprison you. It happens all the time. It’s even happened in America. Just look up DNA exonerations - people have had DNA on the scene that literally proves their innocence and they’re still locked up. People should care about their governments being corrupt, not necessarily about this.

Dah00n 5y ago
Lots of things start out like this, then later they start looking for or blocking other things. Pirated movies, files leaked from three letter agencies, a picture of the president with a ** in his mouth, etc. Even if this is 100% bullet-proof it is still enough that Apple should be seen as privacy invading and leaking everything to the government (and others later on) as it will be abused if implemented everywhere. This isn't irrelevant because of other things happening that are also bad. This is added on-top of those broken system, like the ones in the US you mention.
minitoar 5y ago
The damage is already done by the time it gets to the point of devices being confiscated.
- endisneigh 5y ago
  Can you point to a single instance of that happening where it was due to a false positive?
  - bawolff 5y ago
    In a proposed system that has yet to be deployed?

aaaaaaaaaaab 5y ago

>By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2 n(log n)? I thought the birtday paradox lets you expect collisions after sqrt(n) tries…

yeldarb 5y ago
Different things; birthday paradox was testing them against each other, turning it into a sorting problem was the other.

jeffbee 5y ago

Following the topic because it's interesting, but I'm bothered by the chain of reasoning necessary to reach these conclusions. How sure are we that the means of dumping the model from iOS and converting it to ONNX results in a faithful representation of what NeuralHash actually does?

skygazer 5y ago
FWIW, Apple says that’s not even the NN hash function they’ll be using. This one has been buried in iOS since 2019, so it may have been a prototype. Although dumping the NN image seems straightforward, so this seems a faithful copy of that older hash function. (There’s been a suggestion that it’s sensitive to floating point processor implementations, so small hash differences may occur between different CPU architectures.)
tyingq 5y ago
I agree in theory, but the burden of proof shouldn't be on outsiders that have no other choice than to extrapolate. Proposing something like this running on an end-user's personally owned device should have a really high bar.

stickfigure 5y ago

> it's not obvious how we can trust that a rogue actor (like a foreign government) couldn't add non-CSAM hashes to the list to root out human rights advocates or political rivals. Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Is anyone so naive that they believe the CCP wouldn't deliver such an ultimatum? Apple's position seems to completely ignore recent Chinese history.

tyingq 5y ago
>(like a foreign government) It seems like it wouldn't take that. If you can generate a colliding pair of images, you could probably create a pair where one of the images might get attention with child porn groups and thus, shared around enough to end up in the CSAM database. And where the other was innocuous.
bsdetector 5y ago
> If the CCP says "put these hashes in your database or we will halt all iPhone sales in China", what do you think Apple is going to do? Or maybe China already said "put in this CSAM check or you can't make or sell phones in China". Since Apple's position is contrary to their previous privacy policy and doesn't seem to make a lot of sense, it's quite possible extortion already happened (and not necessarily by China).
- ballenf 5y ago
  I'd honestly believe such pressure came from US domestic intelligence or law enforcement agencies just as easily as from China.
  - adventured 5y ago
    It wouldn't specifically be from domestic intelligence, it would be from a powerful member of Congress with a relationship to Apple (specifically the board/management), acting as a go-between that would try to politically lay out the situation for them. Hey Apple, we can either turn up the anti-trust heat by a lot, or we can turn it down, which is it going to be? Except it would be couched in intellectually dishonest language meant to preserve a veneer that the US Government isn't a violent, quasi-psychotic bully ready to bash your face in if you don't do what they're asking. The interactions with intelligence about the new program would begin after they acquiesce to going along. It's too easy. There's an extraordinary amount of wilful naivety in the US about the nature of the government and its frequent power abuses (what it's willing to do), despite the rather comically massive demonstration of said abuses spanning the entire post WW2 era. Every time it happens the wilfully naive crowd feigns surprise.

pizzaknife 5y ago

the world of forensics is something im not familiar with, a couple questions: - When a hash matches (correctly or incorrectly), how is said image reported? is the matching image passed on to a human to verify or? - what is the survey size / who(m) are subject to this CSAM net?

rootusrootus 5y ago
after ~30 matches, only the matching images are passed on to a human for visual verification. only images uploaded to iCloud are subject to matching
- Dah00n 5y ago
  If they pass CSAM verfied by hash on to human verification inside Apple they break the law. Not even the FBI are allowed to do that. Only NMCEC is an allowed recipient by US federal law.
  - nonbirithm 5y ago
    > Not even the FBI are allowed to do that. The FBI was given clearance to take over a website serving CSAM in order to catch more users of the site. As such, the FBI has technically distributed CSAM in the past. https://www.dallasnews.com/news/crime/2017/01/17/the-fbi-ran-a-child-porn-site-to-catch-predators-and-now-the-accused-are-crying-foul/ https://www.dallasnews.com/news/crime/2017/01/17/the-fbi-ran...
  - rootusrootus 5y ago
    Seems to be a misunderstanding between what the law appears to say and what the actual practice is. Law enforcement's interest is not served by trying to prosecute moderators or companies acting in good faith because they have CSAM in their possession.
    - nybble41 5y ago
      There is a difference between moderators manually identifying illegal content in a stream of mostly-legal material and a process where content which has already been matched against the database and classified as almost-certainly-illegal is subjected to further internal review.

rm999 5y ago

tldr: this is expected. The article addresses everything I'm about to say, but I think the lede is buried: >Apple's NeuralHash perceptual hash function performs its job better than I expected When you are just looking for any two collisions in 100M x 100M comparisons, of course you'll find a small number of positives, Apple said as much. The number of expected collisions will scale linearly with the number of 'bad' images, which is not 100M. Assuming it's 100k, we'd expect 1000x fewer collisions in ImageNet, or ~0.002 collisions, which is effectively 0. It's the artificial images that will potentially sink all this, not a very low rate of naturally occurring hash collisions.

biesnecker 5y ago

> This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. The number of images in that database could well be far in excess of a million. According to NCMEC [1], in 2020 65.4 million files that were reported to them, and "[s]ince the program inception in 2002, CVIP [child victim identification project] has reviewed more than 330 million images and videos." Of course many of those were duplicate but it would be entirely unsurprised if there were more than a million original files. [1] https://www.missingkids.org/ourwork/impact https://www.missingkids.org/ourwork/impact

tehnub 5y ago

> In order to test things, I decided to search the publicly available ImageNet dataset for collisions between semantically different images. I generated NeuralHashes for all 1.43 million images and searched for organic collisions. By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. I don't know what the author means by "taking advantage of the birthday paradox". If they're referring to the "birthday attack" [0], I don't think it makes sense. The birthday attack is a strategy that helps you find a collision without hashing every single image, but he states that he already generated NeuralHashes for all 1.43 million images. Furthermore, isn't there a simple linear time algorithm to detect collisions given that you already have all the hashes? Iterate over your NeuralHashes and put them into a hash table where the NeuralHash is the key, and the number of occurrences is the value. Whenever you insert something into a bucket and there's already something there, you have a neural hash collision. [0]: https://en.wikipedia.org/wiki/Birthday_problem#Probability_table https://en.wikipedia.org/wiki/Birthday_problem#Probability_t...

tibbar 5y ago
1) the point of the birthday paradox is that even if two elements of a set are unlikely to overlap, in a large set it is much more likely than perhaps intuitive that some pair of elements overlap. 2) I’m assuming he did something like putting all the hashes in a list and sorting them, which is at least better than looking at each pair. As you say, it’s not optimal and also doesn’t seem particularly worth including in the otherwise very interesting post
sweezyjeezy 5y ago
By "taking advantage of the Birthday Paradox" - he means that even though the chance of two random images colliding is ~ 1/1 trillion, if you have a set of size ~ sqrt(1 trillion) you have a good chance of having a collision amongst all pairs.
- tehnub 5y ago
  But did the birthday paradox inform his procedure in any way? I guess he wouldn't have even attempted to do this at all if he didn't think it was likely to find collisions.

tbenst 5y ago

I’m much more worried about adversarial attacks rather than accidental false positives. For example, a year ago I was trying to track down how some unscrupulous photos ended up in my photo library. I finally realized that Telegram had, without my knowing permission, been adding all photos I had received in a crypto discussion group to my Apple photo library. There seem many ripe opportunities for targeting.

swiley 5y ago
This kind of thing is why you really can’t have conversations on phones in confidence. You really have no idea what the chat client on your phone (much less the other person’s) is doing.

nanis 5y ago

It sounds like there is code that enables anyone to compute the perceptual hash for an image. Is this code published somewhere?

yeldarb 5y ago
The instructions are here: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX This was discussed on HN yesterday here (and a few other front-page stories): https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391
- nanis 5y ago
  Got it ... Here's the part I was missing: > You will need 4 files from a recent macOS or iOS build: neuralhash_128x96_seed1.dat NeuralHashv3b-current.espresso.net NeuralHashv3b-current.espresso.shape NeuralHashv3b-current.espresso.weights So, for clarity, Apple did not publish the model/code for others to be able to use/test etc. Someone found a clever way to convert the stored model into an open format and therefore others with access to those files are able to experiment.

smithza 5y ago

This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2). Assuming the NCMEC database has more than 20,000 images, this represents a slightly higher rate than Apple had previously reported. But, assuming there are less than a million images in the dataset, it's probably in the right ballpark. If the author was comparing 2 trillion pictures of people, or children specifically, I think this false-positive rate would be different and arguably much higher. The reasons are obvious: humans are similar in dimensions to eachother and are much more likely to match in the same way the hatchet and nematode matched. I do not presume such a finding of photos is easy to come by but I wish the author put details on the sample set.

yeldarb 5y ago
The sample set is ImageNet, which is a well-known dataset in Computer Vision and is available for download here: https://www.kaggle.com/c/imagenet-object-localization-challenge https://www.kaggle.com/c/imagenet-object-localization-challe... I'd love to see this work extended; if you find additional collisions in the wild please submit a PR to the repo (please do not submit artificially generated adversarial images): https://github.com/roboflow-ai/neuralhash-collisions https://github.com/roboflow-ai/neuralhash-collisions For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes) [1] https://tidbits.com/2021/08/13/new-csam-detection-details-emerge-following-craig-federighi-interview/ https://tidbits.com/2021/08/13/new-csam-detection-details-em... > In Apple’s tests against 100 million non-CSAM images, it encountered 3 false positives when compared against NCMEC’s database. In a separate test of 500,000 adult pornography images matched against NCMEC’s database, it found no false positives.
- smithza 5y ago
  For what it's worth, Apple claimed to find a _lower_ incidence of false-positives when it used pornographic images in its test[1] (which makes sense; images containing humans is probably more aligned with what the model was trained on than nematodes) This is an important note. Is it the case that this algorithm is trained for humans or not? the 1/trillion false-positive rate might imply it is trained with a broader set. Thank you for those helpful tidbits.

28194608 5y ago

You guys are helping Apple not to fuckup their implementation by finding bugs. It would be great if these collision posts happened after they fully rolled out the update.

foepys 5y ago
Once the code landed in iOS and is activated, it is too late.

hyperbovine 5y ago

Err, most of these are not naturally occurring pairs since in each case the images differ by a human manipulation (resolution reduction, drawing an arrow, changing the watermark, changing the aspect ratio)---which I'm guessing is viewed as a feature not a bug by the designers of this system. The axe and the nematode comes closest, and even then, they are low-res and visually similar. What would be far more concerning is a hash collision between a picture of a piano and a picture of an elephant, but nothing like that is happening here.

vhold 5y ago

> By taking advantage of the birthday paradox, and a collision search algorithm that let me search in n(log n) time instead of the naive n^2, I was able to compare the NeuralHashes of over 2 trillion image pairs in just a few hours. I think you could just do "sort | uniq -c | sort -nr" on the neuralhash values to find the most frequently occurring ones pretty fast?

occamrazor 5y ago
That’s exactly the O(n log n) algorithm.

kizer 5y ago

Rehash everything using 512 bits or something.

mrtranscendence 5y ago
Would that make a difference? Technically 96 bits is more than enough to assign a unique hash to every image that ever existed, many times over.

oconnore 5y ago

If it's so easy to modify NeuralHashes, won't "CSAM networks" just rotate the hashes of their "collections"? If you can make an innocent picture collide with a CSAM picture, presumably you can also edit a CSAM picture to have a random hash not in the database?

mrtranscendence 5y ago
Through chatting and reading threads I get the impression that many folks think that pedophiles are all smart, extremely tech savvy, operating in networks that share information and collaborate to fool law enforcement. There may be some of that, but there are also plenty of Joe Schmoes who download or trade CSAM material without being especially clever about it. I remember when I was a teenager downloading porn I found more than a few probably-illegal photos just by accident*. If you want to find such images you can, it doesn't take an organized "network". * edit - this was 25 years ago, admittedly, and much of what I downloaded came from randos on AOL. Things might be different now.

tgsovlerkhgsel 5y ago

> Apple now has over 1.5 billion users so we are talking about a large pool of users at stake which increases the likelihood of even a low probability event manifesting This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. What's even worse, when those individuals are facing whatever they're facing, they'll have to argue against the one-in-a-billion odds. If there are jurisdictions where defense lawyers don't get access to the images their client is accused of, and prosecutors and judges don't look at the images but only at the surrounding evidence (which says this person is guilty, with 99.9999999% certainty), Apple may have built a system that's statistically expected to send an innocent person to prison.

mrtranscendence 5y ago
> This is an extremely good point. If the whole system, end to end, after all safeguards (e.g. human reviewers which can also make mistakes) has a one-in-a-billion chance to ruin a user's life, then statistically, we can expect 1-2 users to have their lives ruined. No one will have their lives ruined. If there's a false collision, someone at Apple has to look at the images as a final safeguard. If it's not actually CSAM then it won't ever make it to law enforcement.
- tgsovlerkhgsel 5y ago
  What probability do you ascribe to that reviewer clicking the wrong button, be it out of habit/zoning out (because the system usually shows them true positives), cheating (always clicking "yes" because it's usually correct and allows them to get paid without having to look at horrible images all day), mistake, wrong instructions (e.g. thinking that all images of children, or all porn including adult porn, should be flagged), confusion, or malice? 1 in 10? 1 in 100? 1 in 1000? Pick a number, you now have an estimate of how much room for error there is in the whole system. If you consider 0.15 lives ruined on average per year acceptable, and the reviewers have a 1-in-1000 error rate, then the rest of the system has to make less than 1-in-10-million mistakes per year. And I'm pretty sure 1-in-1000 for the reviewers is very, very optimistic, even if you do quality checks, control for fatigue, etc. On the other hand, hopefully there are some safeguards after the reviewers (e.g. human prosecutors who don't blindly rubber stamp). But my point is: The room for error in anything done at scale that has severe consequences is extremely small.

dan-robertson 5y ago

It didn’t seem to take long for the weights for Apple’s network to be discovered. And I suppose they must send the banned hashes to the client for checking too. So I expect that list will be discovered and published soon too (unless they have some way to keep them secret?) I think one important question is: how reversible is Apple’s perceptual hash? For example, my understanding of Microsoft’s PhotoDNA is that their perceptual hash has been reverse-engineered and that one could go backwards from a hash to a blurry image. But also it is very hard to get the list of PhotoDNA hashes for the NCMEC database. In other words, are Apple unintentionally releasing enough information to reconstruct a bunch of blurry CSAM?

yeldarb 5y ago
It doesn't seem like it; check out the adversarially constructed images here. They don't look anything like the original despite perfectly matching the NeuralHash: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
- dan-robertson 5y ago
  Right obviously the hashing isn’t going to be injective and therefore there are lots of silly images that hash to a given value. The question is whether it is possible to efficiently find plausible images with a given hash. Think more like a “deep dream” than these adversarial attacks.
fortenforge 5y ago
The hashes sent to the client are cryptographically blinded, making it impossible for the client to determine the original CSAM hashes.
- dan-robertson 5y ago
  This is the information I was missing, thanks.
kccqzy 5y ago

saithound 5y ago

Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the article, are both derived without any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the false positive rate is expected to be much much higher. Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar). In an ordinary, non-adversarial context, the probability that you win the game is much less than one in one trillion, so this game is very safe for the casino. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play, and the relevant probability is no longer one in one trillion. The relevant number is 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.

mrtranscendence 5y ago
Why would anyone bother with such an attack? The end result is that some peon at Apple has to look at the images and mark them as not CSAM. You've cost someone a bit of privacy, but that's it.
- e40 5y ago
  > The end result is that some peon at Apple has to look at the images and mark them as not CSAM Btw, this reminded me of a podcast about FB's group to do just this. Because it negatively impacted the mental health of those FB employees, they farmed it out to the contractors in other countries. There were interviews with women in the Philippines, and it was having the same impact there.
  - 5y ago

jb1991 5y ago

> Apple's NeuralHash perceptual hash function performs its job better than I expected and the false-positive rate on pairs of ImageNet images is plausibly similar to what Apple found between their 100M test images and the unknown number of NCMEC CSAM hashes.

Grakel 5y ago

Maybe this is a shot in the arm for anonymous devices. Somebody buy a bunch of GSM devices and resell for cash.

ctur 5y ago

The more interesting point about hash collisions is probably less about accidental clashes and more about intentional clashes. If the hashes in the CSAM database were known publicly, and people began generating intentional clashes with innocuous images, and those images were on many phones, it could basically DOS whatever manual process Apple creates. Basically it could become an arms race where, say, free speech advocates convince people to just keep and share some images and overwhelm the process. Then Apple adapts to new hashes, blocklists known false positives, and the cycle repeats.

joelbondurant 5y ago

Apple employees will do anything to keep their jobs looking nat kiddie porn.

deleted 5y ago

[deleted]

tartoran 5y ago

I am not exactly sure if this is how this works but it appears to me that all the hashes of your photos get uploaded to a server and was wondering if it is possible to reverse the hashes to be able to deduce what's in each photo hashed..

ilogik 5y ago
no and no
- tartoran 5y ago
  Your answer without an explanation doesn't amount to anything more than a scant opinion. "no and no" isn't worth commenting on hackernews in my opinion. As I stated above, I am not very familiar with how this bleeding edge technology works but am not learning anything new from your response. If you wanted to expand I'd appreciate it

cheald 5y ago

The demonstrated ability to produce collisions at will should be an instant show-stopper for this feature. If a bad actor can send targets innocent images which hash to CSAM, you essentially have an upgraded secret swatting mechanism that people will use to abuse others.

kebman 5y ago

Oh boy... I'm just here from watching WEF videos on YouTube. I'm gonna go hide in a cave now.

solidgol 5y ago

I think the headline is misleading. The point of the article is that the algorithm that Apple is using works as well as Apple says it does, and the headline implies that it doesn't.

dang 5y ago
What would be a better (i.e. more accurate and neutral) title?
- solidgol 5y ago
  A reply by the moderator! I am honored. I think the headline should be: ImageNet contains about as many naturally occurring NeuralHash collisions as Apple predicts
  - yeldarb 5y ago
    Hey, OP here - while that is also true, I think the average observer would think this number _should_ be zero. The fact that any exist at all is surprising (but the fact that only two exist is also surprising; I expected to find either zero or lots when I started the script).

orange_puff 5y ago

What we know from this well written and helpful article; The false positive rate Apple told us their algorithm had seems to be accurate. If a machine learning model is extracted from the OS it exists on, it will be much easier to generate adversarial attacks. For example, a neural net's cost function is just a multivariate function with weights as its input. To figure out how to move those weights (positively or negatively), the gradient of the function is calculated and the weights are nudged in the opposite direction (gradient is the direction of the largest growth of a function, we are trying to minimize the cost). Now, assume we are given a cost function and the weights are constant, now, the input can be the image. So, we take the gradient of the cost function with respect to the image pixels and can now see how we should nudge those to maximize the cost. Apple will absolutely need to protect against adversarial attacks for this to be viable. I'm hopeful.

tentacleuno 5y ago

I'm really starting to feel out of touch with Hacker News. Apple seems to be consistently on the front page somehow. Surely other people are sick of hearing about Apple too? They do say any press is good press, though.

GatorD42 5y ago

Wouldn’t the most likely attack be an angry (ex) spouse with physical access to a target’s phone placing images on it (and uploading to iCloud)? I haven’t seen this seemingly likely scenario discussed much.

judge2020 5y ago
two scenarios: - Those images would either have to be actual CSAM (illegal to distribute in the first place and generally hard to get) - fake collision images are placed instead, and all that would do is push the multiple fake CSAM images to Apple's human reviewers who would then likely mark it as a false positive/spam. If the Apple reviewer makes a mistake and does report it as CSAM them the only risk (in the US) is law enforcement having probably cause for search & seizure to look for other CSAM.
- GatorD42 5y ago
  The first scenario seems much more plausible and genuinely worrying.

joemountain 5y ago

From the article, in the conclusion section: "Apple's NeuralHash perceptual hash function performs its job better than I expected" Just want to surface that because the title seems a little sensational based on the conclusion.

IlliOnato 5y ago

> Apple has tried to mitigate this by requiring two countries to agree to add a file to the list, but the process for this seems opaque and ripe for abuse. They mean Russia and Belarus would need to agree to add a file to the list? Yeah, this is a very hard barrier to overcome! /s

NoPicklez 5y ago

As a completely unrelated side note separate to Apple's implementation. I just read today that OnlyFans also contribute hashes to the same organisation, I wonder how many other companies have been doing the same for years. Haven't see any uproar about this elsewhere for other services. For example, CloudFare has had this as an option on their environment for years.

fluidcruft 5y ago

I guess I'd care if any of the collisions weren't of things that I can vaguely see as looking similar. It's not like a nail or a ski looks like child porn. I'm actually impressed. Who cares that they collide. The manufactured collisions are obviously a problem though if I understand correctly that it means CP could be hidden by making it collide with well-known (and likely whitelisted) images.

Threeve303 5y ago

I’m honestly fine with Apple doing this if it’s for the stated reasons. But experience has shown me that this is essentially copyright infringement enforcement dressed up in some emotional blackmail. Of course the point where this became inevitable was when Apple began creating and distributing media.