37 ms·

Hash collision in Apple NeuralHash model

• deleted 5y ago
  [deleted]
• kuu 5y ago
  I don't understand the comment in the issue by an iPhone user. Can you see the hashes that the mobile generates for each image?? Why that is not "obfuscated" / hidden from the user? I mean, I would expect something complicated to validate that you have a collision.
  • nulld3v 5y ago
    They could have a jailbreak device.
    • eptcyka 5y ago
      They could just be a dev that work on the code and have toy apps to test it out.
  • scandinavian 5y ago
    It is hidden for the user. Obfuscation doesn't work. They probably just called the private API to generate a hash on a jailbroken phone. There's even a link to another piece of software that can do just that, only on macOS. https://github.com/KhaosT/nhcalc https://github.com/KhaosT/nhcalc
    • cyanite 5y ago
      The hash table is blinded on the device, and the device never knows if a given image is a hit or not. This is well documented.
  • deleted 5y ago
    [deleted]
  • jhugo 5y ago
    You're holding the iPhone in your hand. You can inspect everything it does, the only thing that varies is the level of difficulty of inspecting it. Obfuscation doesn't work.
  • jdlshore 5y ago
    You're correct. The amount of misinformation in this thread (and in the other responses to you) is out of control. The database of CSAM hashes is blinded and no one has the hashes. Without the hashes, this attack is useless. It's also mitigated by a LOT of checks and balances. First they have to know 30 hashes to target (they're secret). They have to get 30 colliding images on your phone. The images have to be unnoticed by you (why not just infiltrate CSAM, then?) or sufficiently compelling that you don't just delete them. Thirty images have to pass human review at Apple. At least one has to pass human review by law enforcement. Then, and only then, will you be arrested and face a threat. Short version: If somebody wants to frame you for possessing CSAM, there are much easier ways. There is no new threat here. https://xkcd.com/538/ https://xkcd.com/538/
    • cyanite 5y ago
      > The amount of misinformation in this thread (and in the other responses to you) is out of control. True it’s not perfect here, but you should see Reddit, then :p. People there hardly even know what they are mad about.
• deadalus 5y ago
  Expectation : Political rivals and enemies of powerful people will be taken out because c-ild pornography will be found in their phone. Pegasus can already monitor and exfiltrate every ounce of data right now, it won't be that hard to insert compromising images on the infected device. Any news about "c-ild porn" being found on someone's phone is suspect now. This has been done before : 1) https://www.deccanchronicle.com/technology/in-other-news/120117/beware-hackers-are-planting-child-porn-on-peoples-computers.html https://www.deccanchronicle.com/technology/in-other-news/120... 2) https://www.independent.co.uk/news/uk/crime/handyman-planted-child-porn-colleague-s-computer-2037439.html https://www.independent.co.uk/news/uk/crime/handyman-planted... 3) https://www.theatlantic.com/notes/2015/09/how-easily-can-hackers-plant-evidence-contd/405133/ https://www.theatlantic.com/notes/2015/09/how-easily-can-hac... 4) https://www.nytimes.com/2016/12/09/world/europe/vladimir-putin-russia-fake-news-hacking-cybersecurity.html https://www.nytimes.com/2016/12/09/world/europe/vladimir-put... 5) https://www.cnet.com/tech/services-and-software/a-child-porn-planting-virus-threat-or-bad-defense/ https://www.cnet.com/tech/services-and-software/a-child-porn...
  • bko 5y ago
    Isn't it weird how it is weaponized against political enemies but the one person everyone knows did engage in exploitation was protected for decades?
    • newsclues 5y ago
      Selective prosecution.
    • draw_down 5y ago
      More par for the course, I’d say.
    • dijit 5y ago
      I don't think that's weird. Those people are not political enemies, they're allies for whatever regime and their support is important. Better to keep someone powerful in your pocket than to oust them and let someone uncorruptible or uncompromising assume the power vacuum.
• ptidhomme 5y ago
  Yes, just like rape accusations. It doesn't matter that you prove it was false afterwards. Edit : well that was a hint to Assange of course. Probably not true in general. So yes, I mean false accusations.
  • reayn 5y ago
    Why is this getting downvoted? It’s very true, just the accusation of committing such a crime (regardless of whether the person was acquitted or not) can easily ruin many facets of a persons’ life.
    • hosteur 5y ago
      Julian Assange and Jake Appelbaum being prime examples of this.
      • d33 5y ago
        In case of Appelbaum, are there any solid reasons to believe that the accusations are untrue? For Assange, I think that the victim admitted that the accusation was fabricated, isn't that the case?
        • enriquto 5y ago
          > solid reasons to believe that the accusations are untrue this is not how the concept of accusation works
        • coldtea 5y ago
          Accusations must be proven true, not untrue by the accused. And besides the "victim" in the latter case there was a whole lot of diplomatic pressure and political commotion to set him up, with carrots and sticks and the aid of friendly satellite states.
          • _jal 5y ago
            They must be proven true beyond a reasonable doubt to get a person in a funny robe to put them in jail. Normal humans are not required to prove anything in order to think them.
• hughrr 5y ago
  That’s end game. Now you can use it for targeted attacks against innocent people. This needs to be shut down and disposed of immediately. There is no other outcome which is socially acceptable for Apple. I feel vindicated now. There are a lot of people saying that I’m insane as I’ve dumped the entire iOS ecosystem in the last week. But Craig was busy steamrolling out the marketing still only a couple of days back about how this is good for the world. No way am I going back. Edit: I’m going to reply to some child comments here because I can’t be bothered to reply to each one. This is step one down the stairs to hell. If even one person is caught with this it will be leverage to move to step 2. Within a few years your entire frame buffer and camera will be working against you full time.
  • user-the-name 5y ago
    It can't. No actions are taken on hashes alone. The procedure is, if an account uploads some number of images with matching hashes, those images are verified by a human. This can attack that system itself, though, by overloading those humans with too much work looking at random noise, but that requires quite a large organised effort. It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.
    • atrus 5y ago
      I doubt someone who is able/wanting to attack such a system would be unwilling to own some of the material themselves.
    • reuben_scratton 5y ago
      "Verified by a human" - and that human will be an overworked, underpaid, overseas subcontractor who may well have an incentive to mash the "Confirm match" button from time to time to improve his performance.
      • short_sells_poo 5y ago
        Beyond that, are we seriously going to ignore the fact that eventually this system will share the private photos of someone's naked kid with some random subcontractor? How is that even remotely OK? Or that it will find and share actual CSAM with said subcontractor?
• halflings 5y ago
  Apple's scheme includes operators manually verifying a low-res version of each image matching CSAM databases before any intervention. Of course, grey noise will never pass for CSAM and will fail that step. The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. The real challenge is generating a real image that could be mistaken for CSAM at low res + is actually benign (or else just send CSAM directly) + matches the hash of real CSAM. This is why SHAttered [1] was such a big deal, but daily random SHA collisions aren't. [1] https://shattered.io/ https://shattered.io/
  • varispeed 5y ago
    > Apple's scheme includes operators manually verifying a low-res version of each image The reviewer, likely on a minimum wage, will report images just in case. Nobody would like to be dragged through the mud because they didn't report something they thought it is innocent.
  • lifthrasiir 5y ago
    But you can essentially perform DoS attack to human checkers, effectively rendering the entire system grind to a halt. The entire system is too reliant on the performance of NeuralHash which can be defaced in many ways. [1] (Added later:) I should note that the DoS attack is only possible with the preimage attack and not the second preimage attack as the issue seemingly suggests, because you need the original CSAM to perform the second preimage attack. But given the second preimage attack is this easy, I don't have any hope for the preimage resistance anyway. (Added much later:) And I realized that Apple did think of this possibility and only stores blinded hashes in the device, so the preimage attack doesn't really work as is. But it seems that the hash output is only 96 bits long according to the repository, so this attack might still be possible albeit with much higher computational cost. [1] To be fair, I don't think that Apple's claim of 1/1,000,000,000,000 false positive rate refers to that of the algorithm. Apple probably tweaked the threshold for manual checking to match that target rate, knowing NeuralHash's false positive rate under the normal circumstances. Of course we know that there is no such thing like the normal circumstances.
    • 5y ago
• vesinisa 5y ago
  Now this offers Apple a very delicate opportunity to back out of the whole scanning controversy due to technological vulnerabilities.
  • kleene_op 5y ago
    10 bucks they don't.
  • Croftengea 5y ago
    This! Apple almost never admit their mistakes ("You're holding it wrong"), so the technical argument would allow leaving this feature in an indefinite and perhaps opt-in beta.
  • notquitehuman 5y ago
    There’s no going back now. They revealed themselves to be enthusiastic participants in a creeping global surveillance system. They’d have to fire every single executive who passed on an opportunity to tank this thing before they get the opportunity to make the case that they’ve changed.
  • rootusrootus 5y ago
    The only way they will back out is if this misunderstanding of perceptual hashes results in significant blowback from average customers. Apple is certainly not going to be surprised that a perceptual hash collision is technically easy.
  • nonbirithm 5y ago
    The idea that because the state of the art right now is flawed that Apple should give up misses the point. Even if it is flawed, in a few years Apple will just release a new version or a different technique that works more effectively for its stated purpose. Other companies will silently improve the server-side scanning techniques they already use after the public outcry started by Apple blows over. So long as companies feel the need to protect themselves from liability for hosting certain kinds of data, be it criminal or political or moral or anything else, there are no societal checks in place to stop them from doing so. This is not a technological issue like so many people are trying to frame it as; it is a policy issue.
• FabHK 5y ago
  How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database (the NeuralHashes of which, unless I'm mistaken, are not available) 2. Attacker sends that to innocent person 3. Innocent person accepts and stores the picture 4. Actually, need to run step 1-3 at least 30 times 5. Innocent person has iCloud syncing enabled 6. Apple's CSAM detection then flags these, and they're manually reviewed 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times Note that other cloud providers have been scanning uploaded photos for years. What has changed wrt targeted attacks against innocent people?
  • xucheng 5y ago
    > 4. Actually, need to run step 1-3 at least 30 times Depending on how the secret sharing is used in Apple PSI, it may be possible that duplicating the same image 30 times would be enough.
  • the8472 5y ago
    > 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times A better collision won't be a grey blob, it'll take some photoshopped and downscaled picture of a kid and massage the least significant bits until it is a collision. https://openai.com/blog/adversarial-example-research/ https://openai.com/blog/adversarial-example-research/
    • FabHK 5y ago
      Remains to be shown whether that is possible, though.
      • mannerheim 5y ago
        It's been done for image classification.
      • nabakin 5y ago
        Unfortunately, it is very likely to be possible. Adversarial ML is extremely effective. I won't be surprised if this is achieved within the day, if not sooner tbh.
• neximo64 5y ago
  Maybe the best idea is for a sufficient number of people to replicate the hashes with collisions from the CSAM database and make copies of photos with nothing in them like this one and just let Apple deal with it. Maybe it can have text too.
  • cyanite 5y ago
    Best for whom? (Also, we don’t know the exact hash list Apple will use).
• zaptheimpaler 5y ago
  This is so overblown. Scanning images for CSAM seems to be a requirement followed by Facebook, Google, Insta and Snap already [1]: > To put this in perspective, in 2019 Facebook reported 65 million instances of CSAM on its platform, according to The New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000,” Apple, on the other hand, reported 3,000 photos. ALL of those services are already scanning all your photos server side implying complete access to the photo for other purposes. Apple went above and beyond what anyone else does and moved the scanning to be client side! I can't believe they are getting shit for this. They are doing MORE than anyone else to protect your privacy, while still complying with federal laws. If you object to this, you should object to all cloud based photo storage services, and with the federal laws. Apple not only made it more private, but also transparently announced what they do and they are getting shit for it. [1] https://www.engadget.com/apple-child-safety-csam-detection-explainer-183029927.html https://www.engadget.com/apple-child-safety-csam-detection-e...
  • WA 5y ago
    Is it so hard to understand? Some people don’t use cloud storage for precisely the reason that the photos are not encrypted. Now they can’t even use their phone for storing photos. The thing with "only when iCloud is enabled" is only for now. It’s trivial to make Scanning all photos default in a future version.
    • nyuszika7h 5y ago
      That would require a software update and would definitely not go unnoticed. Would you rather they implement scanning on server side and never be able to enable end-to-end encryption for iCloud Photos? I imagine that might be the end goal, otherwise I don't see why they wouldn't have just done it on server side. Sure, this system still has the potential to be abused, but if I had to choose between "end-to-end encrypted with local scanning before upload, which can maybe be abused but has a higher chance of being noticed if it is, and can be disabled with a jailbreak if you're really paranoid" and "not end-to-end encrypted, Apple can inspect my whole photo library whenever they desire, which can go completely unnoticed due to gag orders forcing them to hand data over to governments, not even requiring a software update, and you are completely powerless to do anything about it except for not using iCloud Photos at all", I would definitely choose the former.
• deleted 5y ago
  [deleted]
• borplk 5y ago
  If you piss off the wrong crowd they're going to drop a small bag with white powder in your photo gallery app if you know what I mean. Along with event logs and all. Good luck!
• wisienkas 5y ago
  Not knowing too much of the NeuralHash model, but why are they using MD5 hash, they are known to have many collisions. We don't use MD5 for private/public keys for the same reason
  • enedil 5y ago
    We don't use md5 for private/public keys because md5 is a hashing algorithm, unrelated completely to encryption. Also, what are your reasons to believe that md5 has been used there?
    • Grollicus 5y ago
      Hashes are generally a part of the signature generation used with certificates. See for example "What role do hashes play in TLS/SSL certificate validation?" -> https://security.stackexchange.com/questions/67512/what-role-do-hashes-play-in-tls-ssl-certificate-validation https://security.stackexchange.com/questions/67512/what-role... In certificates, md5 - and sha1 - was used quite some time after it was known to be weak, I suspect OP was thinking of that. This article seems to give a good summary what happened with sha1, mentions md5 in passing and links the related chromium issue: https://konklone.com/post/why-google-is-hurrying-the-web-to-kill-sha-1 https://konklone.com/post/why-google-is-hurrying-the-web-to-...
  • Retr0id 5y ago
    What makes you think they're using MD5 anywhere? Even if they were, it wouldn't matter, because NeuralHash is non-cryptographic by design.
    • wisienkas 5y ago
      I looked at the link and looked at the output from the algorithm for the 2 images which was a MD5 hash. so from that :)
      • Retr0id 5y ago
        But it isn't MD5. It's not even the same length as an MD5 hash. I am confused by your reasoning.
• shadowgovt 5y ago
  Does this algorithm work for the reverse goal (i.e. can content that would trip the CSAM hash be perturbed enough to avoid it without compromising quality of the underlying image)? To my mind, that's far more disquieting than the risk of someone staging and elaborate attack on an enemy's device.
  • zug_zug 5y ago
    Why is this downvoted? This checks out -- A problem with local scanning is that the source code will always be available and thus the user can learn exactly which perturbations trick the system.
• Edd314159 5y ago
  I think I am in dire need of some education here and so I have questions: * Is this a problem with Apple's CSAM discriminator engine or with the fact that it's happening on-device? * Would this attack not be possible if scanning was instead happening in the cloud, using the same model? * Are other services (Google Photos, Facebook, etc.) that store photos in the cloud not doing something similar to uploaded photos, with models that may be similarly vulnerable to this attack? I know that an argument against on-device scanning is that people don't like to feel like the device that they own is acting against them - like it's snitching on them. I can understand and actually sympathise with that argument, it feels wrong. But we have known for a long time that computer vision can be fooled with adversarial images. What is special about this particular example? Is it only because it's specifically tricking the Apple CSAM system, which is currently a hotly-debated topic, or is there something particularly bad here, something that is not true with other CSAM "detectors"? I genuinely don't know enough about this subject to comment with anything other than questions.
  • bo1024 5y ago
    Not complete answers but background: apple’s system works by having your device create a hash of each image you have. The hash (a short hexadecimal string) is compared to a list of known CP image hashes, and if it matches, then your image is uploaded to Apple for further investigation. A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. That appears to be what we are witnessing.
    • magpi3 5y ago
      But how would the attacker get the generated image on a person's phone?
      • Koffiepoeder 5y ago
        E.g. send them a whatsapp message that looks innocent
• newsclues 5y ago
  First CP. Then leaked or unauthorized nudes will get filtered. Filters for terrorists, and terrorist imagery or symbols. Start scanning for guns and drugs. Drug dealers and criminals get added to the list. How long before before it’s dissidents, political opponents, and minorities in dictatorships? How long before Tim Cooks CP filters are used against LGBT groups abroad?
  • Crontab 5y ago
    > How long before before it’s dissidents, political opponents, and minorities in dictatorships? Siri: "You seem to be having unpatriotic thoughts. We are dispatching someone to help you."
    • jnsie 5y ago
      By dispatching someone you presumably mean “ok, I found this on the web”?
  • zug_zug 5y ago
    It's funny how it's always CP or terrorism we need to watch out for.... Never is it the politician who accidentally went to war with wrong country, or the tax return of the congressman, or a cop sleeping with somebody who's under arrest, or EPA employee who took a bribe to declare a chemical safe, or mysteriously malfunctioning cameras in a prison the night of an alleged suicide. Build surveillance to catch those people and I'll listen.
    • newsclues 5y ago
      I am not surprised censorship is used selectively by the powerful against their enemies.
    • personlurking 5y ago
      >It's funny how it's always CP or terrorism we need to watch out for Change the name on the box to something else, but same message. https://i.imgur.com/TAHgUPy.jpg https://i.imgur.com/TAHgUPy.jpg
• UncleMeat 5y ago
  Why is this meaningfully different than, say, what Google Photos has been doing for years? If you can get rooting malware on the target device then you could 1. Produce actual CSAM rather than a hash collision 2. Produce lots of it 3. Sync it with Google Photos This attack has been available for many years and does not need convoluted steps like hash collisions if you have the means to control somebody's phone with a RAT.
  • short_sells_poo 5y ago
    Google was not standing on a pedestal preaching privacy. In contrast Apple was trying to appear as the privacy conscious hardware/software vendor. To now implement such a blatantly obvious stepping stone to dragnet surveillance of actual devices is such a hypocritical move that it beggars belief. Ignoring the whataboutism in your question, we know that privacy once lost is practically impossible to get back. Once the genie is out of the bottle and Apple is doing on device scanning, what's to stop 3 letter agencies and governments around the world to start demanding ever more access? Because that's exactly what's going to happen. "Oh it's not a big deal, we just need slightly more access than we have now." 10 years down the line, we'd have people's phones datamining every piece of info they have and silently reporting to an unknowable set of entities. All in the name of fighting crime. There needs to be pushback on this crap. Every single one of these attempts must be met with absolute and unconditional refusal. Mere inaction means things will inevitably and invariably get worse over time.
    • UncleMeat 5y ago
      It just seems to me that there are two very different conversations happening at the same time, with people swapping back and forth between them 1. There can be false positives or other mechanisms for innocent people to get flagged. 2. It is bad to do this sort of check on the local disk. The discussion at hand started as entirely #1. But now you've swapped to #2, talking about government spying on local files. It makes it very difficult to have a conversation because any pushback against arguments made for one point is assumed to be pushback against arguments made for the other point.
• yayr 5y ago
  How long did it take now to make the Apple algorithm ultimately useless or even harmful? Apple announcement of neural hashing: 5.8.2021. Generic algorithm to generate a different matching image: 8.8.2021. one script was already released 10 days ago here https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29197c0be https://gist.github.com/unrealwill/c480371c3a4bf3abb29856c29...
  • cyanite 5y ago
    None of this makes the system useless or harmful. Also, it’s not Apple’s algorithm. The actual hash list Apple will use is not accessible to the device.
    • yayr 5y ago
      If anybody with enough motivation can modify any existing harmless image to have the same neural hash as a "tracked database" image this will create too many false positives. Too many false positives make the algorithm useless. If someone with even more motivation and the means to put those images onto your device via social engineering, exploits or maybe even features and you become the target of a criminal investigation in any jurisdiction you just happen to be at the moment, this makes the algorithm harmful.
      • FabHK 5y ago
        > this will create too many false positives. Too many false positives make the algorithm useless Apple can (and will) run a second algorithm server side to filter out further false positives. > If someone with even more motivation and the means to put those images onto your device via social engineering, exploits or maybe even features Such an attacker could just plant CSAM directly. The hash collision has no bearing on it. If, however, it is hash collision you're worried about, they'd be caught during the manual review.
• avsteele 5y ago
  Apple claimed 'one in a trillion' chance of a collision. This is a great example of why you should not trust such an assessment.
  • arsome 5y ago
    To be fair, log_2(1 trillion) is only about 40 bits.
  • manmal 5y ago
    That number is based around their plan to only trigger a search if a certain number of CSAM hashes were detected on a phone. A single collision wouldn't trigger such a search. A sufficiently motivated person can of course get access to a phone and put plenty hash-colliding images in the library.
  • cyanite 5y ago
    That’s a different scenario. That’s for it happening by chance, not on purpose. At any rate, the “matching” images are reviewed (or a “visual derivative” is).
• reacharavindh 5y ago
  I admit that I have not done enough research to have a strong opinion on this, but why is Apple taking this on themselves? As far as I can see, this outrage is because of "scanning on iPhone" that is wildly out of user's control. Why can't Apple be like others and say we scan the shit out of what you upload to iCloud(and it is in our Terms and Conditions to use iCloud)? Almost all tech people know that iCloud (or its backups) are not encrypted, so they can decrypt it as they wish.. and those among us that are privacy conscious can comfortable turn iCloud OFF(or not sign up for one) and use the iPhone as a simple private device?
  • jefftk 5y ago
    The speculation that I've seen here is that Apple is rolling this out ahead of a future announcement that iCloud uploads will be encrypted.
    • bitneuker 5y ago
      Are they not? I would've thought that with Apple advertising privacy and security they would at least encrypt iCloud uploads.
      • davidcbc 5y ago
        iCloud uploads are encrypted in transit and on their servers, but they are not E2E encrypted. Apple has the decryption keys
      • reacharavindh 5y ago
        Apple has the decryption keys to iCloud... It is one of those things that many people assume the other way given Apple's marketing and PR stands. I for one am aware of this with iCloud and am still using iCloud for convenience (and by choice). If I were in need of better privacy, I can always use the iPhone without iCloud, and it will work. After this "in phone" watchdog implementation, that will not be the case. I will assume that Apple is constantly watching all my unencrypted content in the worst case, on behalf of state actors and intelligence agencies. We have seen Apple give in to the Chinese government spying because it is the law there. US government could easily ask for this and also apply a gag order preventing Apple from being able to tell the user. The best situation is to lack such a tool.
• magpi3 5y ago
  Can someone ELI5? I understand that a person can now generate an image with the same hash as an illegal image (such as child porn), but I don't understand how they can get it on someone's phone and I don't understand why someone would get in trouble for an image, when finally examined, that is clearly not child pornography.
  • varispeed 5y ago
    > that is clearly not For one, you can't know if that's true as the image could have been manipulated to appear as such. For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image. > but I don't understand how they can get it on someone's phone There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...
    • mukesh610 5y ago
      > For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image. How would NeuralHash pick a "hidden image"? It only uses the pixels of the image to get the hash. Any hidden image in the metadata would not even be picked up and no amount of steganography can fool NeuralHash. > There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc... As iterated elsewhere in this thread, random gibberish pixels colliding with CSAM would definitely not be useful in incriminating anyone. The manual process would catch that. Also, if the manual process is overloaded, I'm pretty sure basic object recognition can filter out most of the colliding gibberish .
      • varispeed 5y ago
        The steganography is not to fool the NeuralHash, but to fool the viewer. The NeuralHash would "see" the planted image, but for the viewer it would appear innocent. I am trying to say that a person reviewing image manually, without special tools will not be able to tell if the image is a false positive and would have to report everything.
• nojito 5y ago
  Any matches are matched again server side to thwart this type of attack. >Once Apple's iCloud Photos servers decrypt a set of positive match vouchers for an account that exceeded the match threshold, the visual derivatives of the positively matching images are referred for review by Apple. First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation. They also fuzz this process by sending false positives I think?
  • notquitehuman 5y ago
    It doesn’t matter what they do after “looking for something to report to law enforcement.” Nothing after that makes it less invasive.
    • floatingatoll 5y ago
      Incorrect. The objections here are threefold: 1) This is invasive. 2) Slippery slope. 3) Gets innocents arrested. Their solution mitigates #3 with review and mitigates #1 with blurring, and those mitigations occur and the point in the process where you claim it doesn’t matter what they do. Please don’t reductively dismiss facts that don’t support your narrative. Saying that it doesn’t matter is wrong and serves only to boost your particular viewpoint on #2.
    • nojito 5y ago
      Disagree. What companies do now is much more invasive. (Indiscriminately scanning cloud storage)
• bugmen0t 5y ago
  Note that a hash collision is not the same as a pre-image attack. Though it seems to me that finding the latter is also feasible.
• topynate 5y ago
  Second preimage attacks are trivial because of how the algorithm works. The image goes through a neural network (one to which everyone has access), the output vector is put through a linear transformation, and that vector is binarized, then cryptographically hashed. It's trivial to perturb any image you might wish so as to be close to the original output vector. This will result in it having the same binarization, hence the same hash. I believe the neural network is a pretty conventional convolutional one, so adversarial perturbations will exist that are invisible to the naked eye. This is useful for two purposes I can think of. One, you can randomize all the vectors on all of your images. Two, you can make problems for others by giving them harmless-looking images that have been cooked to give particular hashes. I'm not sure how bad those problems would be – at some point a police officer does have to look at the image in order to get probable cause. Perhaps it could lead to your Apple account being suspended, however.
  • Kalium 5y ago
    A police raid on a person's home, or even a gentler thorough search, can be enough to quite seriously disrupt a person's life. Certainly having the police walk away with all your electronics in evidence bags will complicate trying to work remotely. Of course, this is assuming everything works as intended and they don't find anything else they can use to charge you with something as they search your home. If you smoke cannabis while being in the wrong state, you're now in several more kinds of trouble.
    • topynate 5y ago
      This could happen with a perturbed image, but I doubt it. Apple will send the suspicious images to the relevant authorities. Those authorities will then look at the images. The chances are low that they will then seek a search, even though the images are innocent upon visual inspection. But maybe in some places a ping from Apple is good enough for a search and seizure.
      • Kalium 5y ago
        In broad strokes, I agree with you. I think you're absolutely correct that most trained, educated, technologically sophisticated law enforcement bodies will do exactly as you suggest and decide that there's not enough to investigate. That said, I'm not willing to say it won't happen. There are too many law enforcement entities of wildly varying levels of professionalism, staffing, and technical sophistication. Someone innocent, somewhere, is likely to have a multi-year legal drama because their local PD got an email from Apple. And we haven't even gotten to subjects like how some LEOs will happily plant evidence once they decide you're guilty.
• sandworm101 5y ago
  Every bad day for Apple is another great day for Linux. Smug mode activated.
• orange_puff 5y ago
  Today HackerNews learns about adversarial machine learning, which every ML model is susceptible to.
• tambourine_man 5y ago
  Neuralhashes are far from my area of expertise, but I've been following Apple closely ever since its foundation and have probably watched every public video of Craig since the NeXT take over and here is my take: I've never seen him so off balance before as in his latest interview with Joanna Stern. Not even in the infamous “shaking mouse hand close up” of the early days. Whatever you say about Apple, they are an extremely well oiled communication machine. Every C-level phrase has a well thought out message to deliver. This interview was a train wreck. Joanna kept asking: please, in simple terms, to a hesitant and inarticulate Craig. It was so bad that she had to produce infographics to fill the communication void left by Apple. They usually do their best to “take control” of the narrative. They were clearly caught way off guard here. And that's revealing.
  • cwizou 5y ago
    I think they clearly didn't anticipate that people would perceive it as anything but a breach of trust, that their device was working against them (even for a good cause, against the worst people). And because of this they calibrated their communication completely wrong, focusing on the on device part as being more private. Using the same line of thinking they use for putting Siri on device. And the follow up was an uncoordinated mess that didn't help either (as you rightly pointed out with Craig's interview). In the Neuenschwander interview [1], he stated this : > The hash list is built into the operating system, we have one global operating system and don’t have the ability to target updates to individual users and so hash lists will be shared by all users when the system is enabled. This still has me confused, here's my understanding so far (please feel free to correct me) - Apple is shipping a neural network trained on the dataset that generates NeuralHashes - Apple also ships (where ?) a "blinded" (by an eliptic curve algo) table lookup that match (all possible?!) NeuralHashes to a key - This key is used to encrypt the NeuralHash and the derivative image (that would be used by the manual review) and this bundle is called the voucher - A final check is done on server using the secret used to generate the elliptic curve to reverse the NeuralHash and check it server side against the known database - If 30 or more are detected, decrypt all vouchers and send the derivative images to manual review. I think I'm missing something regarding the blinded table as I don't see what it brings to the table in that scenario, apart from adding a complex key generation for the vouchers. If that table only contained the NeuralHashes of known CSAM images as keys, that would be as good as giving the list to people knowing the model is easily extracted. And if it's not a table lookup but just a cryptographic function, I don't see where the blinded table is coming from in Apple's documentation [2]. Assuming above assumptions are correct, I'm paradoxically feeling a tiny bit better about that system on a technical level (I still think doing anything client side is a very bad precedent), but what a mess did they put themselves into. Had they done this purely server side (and to be frank there's not much difference, the significant part seems to be done server side) this would have been a complete non-event. [1] : https://daringfireball.net/linked/2021/08/11/panzarino-neuenschwander-interview https://daringfireball.net/linked/2021/08/11/panzarino-neuen... [2] This is my understanding based on the repository and what's written page 6-7 : https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
• rootsudo 5y ago
  And everyone was saying how impossible this was. Not an expert, but I do enjoy how quickly this was absconded and disproven.
  • Ajedi32 5y ago
    Impossible (or nearly so) for a cryptographically secure hash function, not a perceptual hash. When the stories on this first broke it wasn't entirely clear which type of hash Apple was using for their CSAM detection. (I assume now we know they're using NeuralHash?)
• guywhocodes 5y ago
  I was on the fence on this one because of Alex Stamos statements on this. I did expect to be able to trust him and in extension the Stanford Internet Observatory more.
• Traubenfuchs 5y ago
  Nothing about this ever made any sense. Any pedophile with an IQ over room temperature already deleted their iCloud at this point, with all the media buzz about it.
• rogers18445 5y ago
  The hash was supposed to not be altered by minor perturbations to the input. This requirement assures that the hash can never be cryptographically secure, collision attacks therefore are trivial the moment the "security by obscurity" aspect is broken.
• raspasov 5y ago
  I'm not in favor of assuming that everyone's guilty until proven innocent. But, as a side note... I get the feeling that a lot of people assume that the CSAM hashes are going to be stored directly on everyone's phone so it's easy to get a hold of them and create images that match those hashes. That does not seem to be the case. The actual CSAM hashes go through a "blinding" server-side step. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  • cyanite 5y ago
    It seems to me that the vast majority of people in forums don’t actually know how this system is implemented. Although people definitely seem more informed here than on Reddit.
• pthrowaway9000 5y ago
  Told ya. Sent from my S5 running lineageOS since 2015, now with MicroG.
• HumblyTossed 5y ago
  This can also be used to make Apple's system useless, no? If enough (millions?) of people were to, say, go to a web site and save generated gray-blobs to their phones, it would create enough false-positives to kill this system, right? Maybe game it and have everyone convert their various profile pics to these images.
  • floatingatoll 5y ago
    No, because Apple will simply start killing Apple IDs and blocking hardware by serial number for abusive behavior towards Apple when y’all do that, and that’s a very expensive problem to overcome. “You tried to exploit our production systems and we’re ending our customer relationship with you over it” is a classic Apple move and no one outside of the Hackintosh community realizes that their devices include crypto-signed attestations of their serial number during Apple service sign-ins.
  • robertoandred 5y ago
    And how would anyone know that those gray blobs match child porn?
    • jug 5y ago
      Exactly, that’s the scary abuse scenario where people could send a whole load of this stuff to e.g a political enemy.
      • robertoandred 5y ago
        What? Why do you think gray blobs would hurt anyone?
        • dannyw 5y ago
          No one is going to send gray blobs, they will be finding legal porn (like pussy close ups, tongue pics, whatever) and then disturbing it to trigger a CSAM hit. The low res derivative will match, perhaps even closely, because pussy closeups look similar to an apple employee when its grayscale 64 by 64 pixels (remember: it's illegal for Apple to transmit CSAM, so it must be so visually degraded to the point where it's arguably not visual). The victim will get raided, be considered a paedophile by their workplace, media, and family, and perhaps even go into jail. The attacker in this case can be users of Pegasus unhappy with a journalist.
• farmerstan 5y ago
  Apple should be embarrassed for itself. The “one in trillion hash” was cracked in a week. Everyone associated with that should be fired.
• rollinggoron 5y ago
  How is this scenario unique to Apple but not everyone else who does scanning? e.g. Google, Facebook, Microsoft etc...
  • floatingatoll 5y ago
    It’s not. The hacker community just never knew or cared about CSAM scanning before Apple did it.
  • steeleduncan 5y ago
    If the entity doing the scanning has a copy of the original image they can verify it is illegal before calling the police. With Apple's system they have to call the police on the basis of the image hash without verifying that anything illegal is on the phone. You can whatsapp someone an innocent image doctored to have a hash collision with known CSAM. If they have default settings it will be saved to their photo reel, scanned by iOS and the police will be called. Until the arresting police officer explains to them they are being arrested on suspicion of being a paedophile, they won't even know this has happened.
    • ec109685 5y ago
      A low res copy is available to the reviewer on Apple's end to verify.
    • cyanite 5y ago
      Apple doesn’t “call the police”, though, they contact the center for child abuse or whatever it’s called, who will then presumably verify the picture.
      • dragonwriter 5y ago
        The National Center for Missing and Exploited Children, a “private” nonprofit created by the US government and primarily funded by the Department of Justice.
• deleted 5y ago
  [deleted]
• proactivesvcs 5y ago
  Many of us have been at the sharp end of a large company's supposed "human review" process when it comes to account lockouts, invalid abuse reports and invalid accusations of breach of terms of service. It simply does not work for too many and leaves us with little or no redress; it is a completely untenable fallback. When you're promised that this will never happen to you because there are humans in the process remember that they're one or more of underpaid, poorly-trained, poorly-treated, under-resourced or just flat-out terrible at their job. This may well happen to you and you'll have no way back.
• dgb99 5y ago
  can someone upvote me please? I need to add my account to my keybase profile.
• nine_k 5y ago
  I think this is a great, high-profile example of how keeping things open helped quickly identify a critical problem. AFAICT the hash function / perceptual model is openly available, or, at least, easy to find online.
• SXX 5y ago
  Some people here in comments believe that whoever gonna check reported material on Apple side will never ever flag false-positive. We already know that NCMEC database itself don't exclusively contain child porn, but also some other photos that closely related ot CSAM. Even if those photos don't have actual CSAM on them. But let's ignore this fact. Do people who believe in behevolent Apple understand that CSAM don't all have some big red "CHILD PORN" sign on it? No demonic feel included. Like any porn we can suppose that many of such images might not even have any faces or literally anything that make them different from images of 18yo person. When you think well about it brute-forcing actual porn or some jailbait images doesn't sound that impossible. All you need is a lot of totally legal porn and some compute power. Both we have in abundance.
  • dannyw 5y ago
    I guarantee you out of millions of alleged CSAM images hapzardly added by local police and intetnet reports, thousands are legal porn of consenting adults.
    • SXX 5y ago
      I totally sure it's very much possible, but even if this database only exclusively contained CSAM it's still very much possible for human to match false-positive since Apple can only show their snoops your photos and likely they will be compressed to some 360x360 or whatever.
  • int_19h 5y ago
    Quite frankly, I don't even care whether the reviewers are 100% perfect or not. I did not consent to having some random stranger out there reviewing my private photos. Even if it doesn't actually have any legal consequences, it's still unacceptable.
• newArray 5y ago
  Hash collisions happen by design in the perceptual hash, its supposed to give equal hashes for small changes after all. Something I find interesting is the necessary consequences of the property of small edits resulting in the same hash. We can show that this is impossible to absolutely achieve, or in other words there must exist an image such that changing a single pixel will change the hash. Proof: Start with 2 images, A and B, of equal dimension, and with different perceptual hashes h(A) and h(B). Transform one pixel of A into the corresponding pixel of B and recompute h(A). At some point, after a single pixel change, h(A) = h(B), this is guaranteed to happen before or at A = B. Now A and the previous version of A have are 1 pixel apart, but have different hashes. QED We can also ATTEMPT to create an image A with a specified hash matching h(A_initial) but which is visually similar to a target image B. Again start with A and B, different images with same dimensions. Transform a random pixel of A towards a pixel of B, but discard the change if h(A) changes from h(A_initial). Since we have so many degrees of freedom for our edit at any point (each channel of each pixel) and the perceptual hash invariant is in our favor, it may be possible to maneuver A close enough to B to fool a person, and keep h(A) = h(A_initial). If this is possible one could transform a given CSAM image into a harmless meme while not changing the hash, spread the crafted image, and get tons of iCloud accounts flagged.
  • mrits 5y ago
    The best part of this is we could all be flagged in a database for things like this and not even realize it. Who knows what kind of review process the governments are actually keeping around these
• a-dub 5y ago
  it gets even more fun when you can generate images that are imperceivably different to the human eye, but produce different (and controlled) hash values. (see shazam decoys and adversarial images). not sure if collision is really the right term here though as similarity hashing intentionally hashes multiple images to the same hash. in other news, my google phone just helpfully scanned all client side images, rounded up everything it thought was a meme and suggested i delete them to save space. i wonder if that feature has telemetry built in...
• marcob95 5y ago
  Can someone help me figure out why this is such a big deal? Thanks
  • cyanite 5y ago
    I’d say that it’s not, and that many people think it is is because they haven’t fully read or understood the technical description of the system. But what you should do is read the documents yourself.
• bengale 5y ago
  I don't understand why this is a concern. Someone would need 30 or so child porn images to generate these. Then they'd need to create these grey blobs and send them to the target. The target would need to save them in their photo library for some reason so they sync to iCloud. Then when all this triggers the review they'll see they are grey blobs and nothing else will happen?
  • teitoklien 5y ago
    Its astonishing , How society went from keeping personal photos private , to thinking its ok , to let a random apple employee scan their photos for suspicion and debate on how human reviewers from a faceless corporation can easily stop these mistakes there. I’d trust the algorithm more than the human reviewing it , considering the algorithm already shows flaws and loopholes , the human part of it does not bring any confidence. This isnt facebook , this is your private photos , it shouldnt have reached till this stage , in the first place
    • bengale 5y ago
      But they won't be my private photos, the only way it gets to the point where a human see's them is that I have a bunch of child porn uploaded into iCloud, or someone puts a bunch of these grey blob images into my library. Neither of those are my images. There is no chance 30+ of my personal images have hash collisions with this database of child porn. > How society went from keeping personal photos private Also, didn't the local photo development shops used to call the police quite frequently when dodgy images were sent in to be developed? I remember it happening once at our local supermarket.
      • teitoklien 5y ago
        Hash collisions dont need grey images You can take a perfectly normal image , play around with a range of its individual pixels, to get collisions too. So someone might forward you your personal pics from your meeting with them and its pixels might get edited by the messenger app you use to save the picture into your photos to specifically collide its hash with a csam image, while to you the image will look perfectly normal This is one example , lets say you 100% trust every developer of every app that you download on apple’s phone. Fine, but that’s already a lot of trust on a lot of people about something that can ruin your life. Do you now trust every single image which might get auto downloaded to your gallery by a malicious actor , what if a random anon person messages you with 100 such colliding images , enough to cross threshold to get the authorities knocking your door , Is this “feature” really worth it , to have the inconvenience of authorities knocking your door and going through legal trouble ? For an iphone ?
• jakear 5y ago
  Baseless speculation: this is less about ongoing protection against keeping CSAM on their servers and more about a one-time sting operation requested by some sort of acronym’d official. The basic idea being: have a bunch of catfish agents send out these known CSAM images to folks through anonymous channels (sourcing the targets would likely be unfortunately trivial), expect that some of them will save the images and be synced to iCloud, coerce Apple into letting them see who has the material, sting them. To this end, I would not be at all surprised to see that in some not-too-distant future Apple issues a big ol’ public apology and removes this feature. The operation will of course be long complete by then. Just writing this out here so I have a “I told you so” link for if/when that time comes :)
• yeldarb 5y ago
  I was curious how big of a deal this would be in a real-world attack (eg could someone use this to DDoS Apple's human reviewers?) so I ran the colliding images through another feature extraction model, OpenAI's CLIP, to see if they also fooled it. They don't; I think it'd be much harder to create an image that both matches the NeuralHash of a CSAM image and also fools a generic model like CLIP as a sanity check for being a CSAM image. I wrote up my findings here: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/
  • dannyw 5y ago
    I heard that a lot of content on the CSAM database don't individually look like child porn. For example, a vaginal close up that is of a 17 year old, but could look like a vaginal close up of a 19 year old. Remember, anything under 18 is treated the exact same way from a legal perspective, and once discovered Apple must report. So there is an adversarial attack in that you find legal porn of say 18 year old pussy closeups, disturb it to match CSAM with the neural hash, and send it to your enemies. The Apple employee will verify a match. The Feds will raid your target and imprison them, and reputationally tarnish them for life. Now think about how Pegasus can remotely modify your photos. And think about Julian Assange's sexual assault allegations that were admitted to be fabrications by the accuser.
• pkulak 5y ago
  If your first thought, like mine, was "Who cares? The whole point is that there will be plenty of false positives.", this is pre-image: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-901016864 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
  • fortenforge 5y ago
    I don't think that's true. This was a second pre-image attack.
  • fogof 5y ago
    That was, indeed, my first thought. Can you explain why reposting the link that the OP posted is supposed to change my mind, or how it contributes to the conversation?
• azinman2 5y ago
  It’s not enough to have collisions — you’ll have to have collisions against ANOTHER hidden model for the same photos, and then have it pass a human looking at it. People are getting unnecessarily angry over this.
• smlss_sftwr 5y ago
  "it's ok, there's only a 1 in a trillion chance of this happening"
• maz1b 5y ago
  No longer considering purchasing any more Apple devices or services for any team members in my company. This is unacceptable.
• th0ma5 5y ago
  Just visiting this page would trigger a bit right?
• daralthus 5y ago
  Putting aside the original intent what else can they scan for that's worth doing in the aggregate with this tech?
• sierpinsky 5y ago
  "According to media reports, the cloud computing industry does not take full advantage of the existing CSAM screening toolsto detect images or videos in cloud computing storage. For instance, big industry players, such as Apple, do not scan their cloud storage. In 2019, Amazon provided only eight reports to the NCMEC, despite handling cloud storage services with millions of uploads and downloads every second. Others, such as Dropbox, Google and Microsoft perform scans for illegal images, but 'only when someone shares them, not when they are uploaded'." [1] So I guess the question is what exactly "others" are doing, 'only when someone shares them, not when they are uploaded'. The whole discussion seems to center around what Apple intends to do on-device, ignoring what others are already doing in the cloud. Isn't this strange? [1] https://www.europarl.europa.eu/RegData/etudes/BRIE/2020/659360/EPRS_BRI(2020)659360_EN.pdf https://www.europarl.europa.eu/RegData/etudes/BRIE/2020/6593...
  • cyanite 5y ago
    > The whole discussion seems to center around what Apple intends to do on-device, ignoring what others are already doing in the cloud. Isn't this strange? Very strange. Especially when this on-device technique means that Apple needs to access far less data than when doing it on the cloud.
  • eightysixfour 5y ago
    It is a shift in trust. If my things are scanned on my local device, I now must trust that Apple: * Will not be compelled to change the list of targeted material by government coercion. * Will not upload the "vouchers" unless the material is uploaded to iCloud. * Will implement these things in such a way that hackers cannot maliciously cause someone to be wrongly implicated in a crime. * Will implement these things in such a way that hackers cannot use the tools Apple has created to seek other information (such as state sponsored hacking groups looking for political dissidents). And for many of us, we do not believe that these things are a given. Here's a fictional scenario to help bring it home: Let's say a device is created that can, with decent accuracy, detect drugs in the air. Storage unit rental companies start to install them in all of their storage units to reduce the risk that they are storing illegal substances, and they notify the police if the sensors go off. One storage unit rental company feels like this is an invasion of privacy, so they install the device in your house but promise to only check the results if you move your things into the storage unit. This sounds crazy, right?
• dang 5y ago
  Ongoing related threads: Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706 https://news.ycombinator.com/item?id=28225706 Convert Apple NeuralHash model for CSAM Detection to ONNX - https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391
• visarga 5y ago
  Have you considered that Apple might have a second neural hashing network which is hidden and they let this one network be hacked to throw people off?
  • faeriechangling 5y ago
    What a genius publicity move
• spoonjim 5y ago
  Any idea why Apple had to be too cute by half and not just scan these files server-side? Or why it doesn't change their plan to do that, given the backlash?
  • cyanite 5y ago
    I hope not, since doing the scans like this offers much more privacy for the user. Even when the user is too ignorant (sorry) to realize this. But this will be evident by reading and understanding the technical description.
    • spoonjim 5y ago
      I'm not concerned with this system -- I'm concerned with the things that governments will now start to force Apple (and any computer vendor) to start scanning for on the client side. Once the client side scanning Rubicon is crossed, countries will want vendors to scan everything (not just iCloud folders) for everything (not just CSAM).