18 ms·

Show HN: Neural-hash-collider – Find target hash collisions for NeuralHash

623 points by anishathalye 5y ago

anishathalye 5y ago
The README (https://github.com/anishathalye/neural-hash-collider#how-it-works https://github.com/anishathalye/neural-hash-collider#how-it-...) explains in a bit more detail how these adversarial attacks work. This code pretty much implements a standard adversarial attack against NeuralHash. One slightly interesting part was replacing the thresholding with a differentiable approximation. I figured I'd share this here in case anyone is interested in seeing what the code to generate adversarial examples looks like; I don't think anyone in the big thread on the topic (https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...) has shared attack code for NeuralHash in particular yet.
- IncRnd 5y ago
  Nicely done. Thank you for sharing. I'd like to share the following paper for anyone else who may be interested. It is about watermarking rather than a preimage attack. "Adversarial Embedding: A robust and elusive Steganography and Watermarking technique" https://arxiv.org/abs/1912.01487 https://arxiv.org/abs/1912.01487 Unfortunately, the existence of invisible watermarking demonstrates a separate attack on the hash. Instead of a preimage attack, this might be able to change the hash of an image that is suspected of already being a match. A true-positive would be changed into a false-negative.
- GistNoesis 5y ago
  Well done ! Here is my version that uses Scipy lbfgs-b optimizers : https://gist.github.com/unrealwill/d64d653a7626b825ef332aa3b2aa1a43 https://gist.github.com/unrealwill/d64d653a7626b825ef332aa3b...
Banditoz 5y ago
That was scary fast. Is there a point in using this algorithm for its intended purpose now?
- dannyw 5y ago
  If the intended purpose is to lead by example and eventually mandate code on every computing device (phone and computer) that scans all files against a government provided database, then yes, that purpose still exists and this algorithm still works for it. Just wait and watch - I guarantee you that Apple will be talking about CSAM in at least one anti-trust legal battle about why they shouldn't be broken up. Because a walled garden means they can oppress citizens on behalf of governments better.
- zepto 5y ago
  Yes, because isn’t a weakness in the design. There is nothing scary fast about it. It was obvious and anticipated in the threat model.
seph-reed 5y ago
Is it possible to host this online as a meme filter? I think every meme should get pumped through this, just for lulz.
m2com 5y ago
Hi, this looks interesting but I have no idea what this all means lol Is this some way of hiding a picture within a picture, or am I way off the mark?
- sodality2 5y ago
  Apple began scanning for CSAM with Neuralhash. This allows you to turn an image into a specific neuralhash thus possibly triggering its (automatic) CSAM detection. Imagine if a picture of a cat could cause Apple to think you have CP on your device.
  - m2com 5y ago
    Oh damn, that’s crazy. Very cool project. Thanks for sharing and the explanation.
  - neom 5y ago
    Well, you'd have to do it 30 times to trigger the system, and then someone at apple moderation would look at those 30 pictures of cats and hit "next" vs "supervisor"
    - m2com 5y ago
      Good that there’s some human supervision. But, I know I have more than 30 photos of my dog. Also don’t like the idea of false positives auto-sharing some of my camera roll.
      - neom 5y ago
        It's only if you back it up to iCloud, the signatures of the CP used as references are rotated, and they're also not public. The chances of you randomly triggering the system is effectively 0 unless you're uploading CP to your iCloud.
        Alupis 5y ago
        Wait, wasn't all the hullabaloo over this scanning not requiring an upload to iCloud anymore? They're scanning anything you upload to iCloud (and have been for some time) but now also scan everything on your device too.
ryanmarsh 5y ago
Ok so now all we have to do is get a phone, load it with adversarial images that have hashes from the CSAM database and we wait and see what happens. Basically a honeypot. Get some top civil rights attorneys involved. Take the case to the Supreme Court. Get precedence set right. Lawfare
- robertoandred 5y ago
  Where would you get the CSAM hashes?
  - dannyw 5y ago
    Give it a few days, and you'll probably find someone selling a list of CSAM neural hashes on darknet marketplaces.
    - Dylan16807 5y ago
      Or tweeting out a bunch of them. They're just 12 byte numbers.
      - copperx 5y ago
        I bet there's a list of hashes already up in Pastebin.
  - arsome 5y ago
    The client has to be able to check for them in some way - just run that algorithm against every image you can scrape from Tor/Freenet and I suspect you'll have results rather quickly. Or you can probably just wait a minute and pay an... enterprising individual to sell you such a list on a darknet market though, or perhaps even find one posted on the clearnet soon enough.
    - robertoandred 5y ago
      No, the client doesn’t have access to the CSAM hashes. And matches are verified on the server, not on the client.
deleted 5y ago
[deleted]
ncw96 5y ago
Apple has said this is not the final version of the hashing algorithm they will be using: https://www.vice.com/en/article/wx5yzq/apple-defends-its-anti-child-abuse-imagery-tech-after-claims-of-hash-collisions https://www.vice.com/en/article/wx5yzq/apple-defends-its-ant...
- only_as_i_fall 5y ago
  Does it matter? Unless they're going to totally change the technology I don't see how they can do anything but buy time until it's reverse engineered. After all, the code runs locally. If Apple wants to defend this they should try to explain how the system will work even if generating adversarial images is trivial.
  - ncw96 5y ago
    Apple has outlined[1] multiple levels of protection in place for this: 1. You have to reach a threshold of matches before your account is flagged. 2. Once the threshold is reached, the matched images are checked against a different perceptual hash algorithm on Apple servers. This means an adversarial image would have to trigger a collision on two distinct hashing algorithms. 3. If both hash algorithms show a match, then “visual derivative” (low-res versions) of the images are inspected by Apple to confirm they are CSAM. Only after these three criteria are met is your account disabled and referred to NCMEC. NCMEC will then do their own review of the flagged images and refer to law enforcement if necessary. [1]: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
    - Dylan16807 5y ago
      I do want to note that decrypting the low-res images would have to happen before step 2.
    - copperx 5y ago
      Will the high-resolution images be collected and used as evidence? Or just the visual derivatives? That's not clear.
jchw 5y ago
I am strongly against Apple’s decision to do on-device CSAM detection, but: wasn’t there a secondary hash whose database is not shared? In theory you need to collide with both to truly defeat the design, right?
- PBnFlash 5y ago
  You just need a sample of something that is evidently so pervasive we're building a nation wide dragnet to stop.
  - jchw 5y ago
    I doubt finding it would really be that hard, if you wanted to be on a list somewhere in some government database, but even armed with a full image that is in the NCMEC database, the problem is that the second hashing algorithm runs on the server and presumably has secure-by-obscurity details… so it would be hard to collide with it on purpose unless you are an insider. That’s my understanding, although details have been a bit shaky at times.
- only_as_i_fall 5y ago
  I don't see how. They're hashing on feature space (so trivial cropping and such doesn't defeat this) but they have two totally separate methods of matching those hashes? Doesn't sound right to me...
  - jchw 5y ago
    Apparently the images in question would get sent to the server, and all calculation happens there. > In a call with reporters regarding the new findings, Apple said its CSAM-scanning system had been built with collisions in mind, given the known limitations of perceptual hashing algorithms. In particular, the company emphasized a secondary server-side hashing algorithm, separate from NeuralHash, the specifics of which are not public. If an image that produced a NeuralHash collision were flagged by the system, it would be checked against the secondary system and identified as an error before reaching human moderators. https://www.theverge.com/2021/8/18/22630439/apple-csam-neuralhash-collision-vulnerability-flaw-cryptography https://www.theverge.com/2021/8/18/22630439/apple-csam-neura... For one reason or another Apple really wants to create this precedent, so it’s only natural they’re doing every last thing to make the feature hard to defeat.
robertoandred 5y ago
…and? Does OP think reviewers will think a picture of a cat is CSAM?
- tibbar 5y ago
  The input image is a parameter, so you could start with some image that would be easier to confuse at the low resolutions the reviewers use. The missing piece in this case is that the database of target hashes is unknown.
  - Dylan16807 5y ago
    Easier to confuse, or even something like a pic of a 20 year old where it's effectively impossible to be sure from the image itself.
Scaevolus 5y ago
NeuralHash collisions are interesting, but the way Apple is implementing their scanner it's impossible to extract the banned hashes directly from the local database. There are other ways to guess what the hashes are, but I can't think of legal ones. > Matching-Database Setup. The system begins by setting up the matching database using the known CSAM image hashes provided by NCMEC and other child-safety organizations. First, Apple receives the NeuralHashes corresponding to known CSAM from the above child-safety organizations. Next, these NeuralHashes go through a series of transformations that includes a final blinding step, powered by elliptic curve cryptography. The blinding is done using a server-side blinding secret, known only to Apple. The blinded CSAM hashes are placed in a hash table, where the position in the hash table is purely a function of the NeuralHash of the CSAM image. This blinded database is securely stored on users’ devices. The properties of elliptic curve cryptography ensure that no device can infer anything about the underlying CSAM image hashes from the blinded database. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- dannyw 5y ago
  You can extract the hashes with a few hours spent on the darknet. Doing that is certainly illegal and not to mention VERY morally wrong, but criminals exist and criminals won't hesitate to abuse this as a mechanism for framing, extortion, or ransom. It's also possible for someone (Attacker A) to go on the darknet and get a list of 96-bit neural hashes, and then publish or sell this list somewhere to another party, Attacker B. The second party would never have to interact with CSAM. Imagine Ransomware v2: We have inserted 29 photos of CSAM-matching material into your photo library. Pay X monero to this address in 30 minutes, or we will insert 2 additional photos, which will cross the threshold and may result in serious and life-changing consequences to you[1]. The difference here (versus the status quo) is that an easily-broken perceptual hashing enables the attacker to never send or possess any CSAM images[2]. From my experiences with being victims of various hackers, I know a lot of them won't touch CSAM because they know it's wrong, but they'll salivate at an opportunity to weaponise automated CSAM scanning. [1]: If you think Apple's human review will mitigate this attack, you can permute legal pornography to match CSAM signatures. If Apple's reviewers see 30 CSAM matches and the visual derivatives look like porn, they will be legally required to report to to NCMEC (a statutory quasi-government agency staffed by the FBI), even if all the photos are actually consensual adults. [2]: If you never possess nor touch CSAM, it might be harder for you to get charged with CP charges. You might be looking at CFAA, blackmail or extortion charges; while your victim faces child pornography charges. This is basically an "amplification attack" on the real world judicial system.
onepunchedman 5y ago
This is just getting wilder and wilder by the day, how spectacularly this move has backfired. As others have commented, at this point all you need is someone willing to sell you the CSAM hashes on the darknet, and this system is transparently broken. Until that day, just send known CSAM to any person you'd like to get in trouble (make sure they have icloud sync enabled), be it your neighbour or a political figure, and start a PR campaign accusing the person of being investigated for it. The whole concept is so inherently flawed it's crazy they haven't been sued yet.
- deleted 5y ago
  [deleted]
- dannyw 5y ago
  The "send known CSAM" attack has existed for a while but never made sense. However, this technology enables a new class of attacks: "send legal porn, collided to match CSAM perceptual hashes". With the previous status quo: 1. The attacker faces charges of possessing and distributing child pornography 2. The victim may be investigated and charged with child pornography if LEO is somehow alerted (which requires work, and can be traced to the attacker). Poor risk/reward payoff, specifically the risk outweighs the reward. So it doesn't happen (often). --- With the new status quo of lossy, on-device CSAM scanning and automated LEO alerting: 1. The attacker never sends CSAM, only material that collides with CSAM hashes. They will be looking at charges of CFAA, extortion, and blackmail. 2. The victim will be automatically investigated by law enforcement, due to Apple's "Safety Voucher" system. The victim will be investigated for possessing child pornography, particularly if the attacker collides legal pornography that may fool a reviewer inspecting a 'visual derivative'. Great risk/reward payoff. The reward dramatically outweighs the risk, as you can get someone in trouble for CSAM without ever touching CSAM yourself. If you think ransomware is bad, just imagine CSAM-collision ransomware. Your files will be replaced* with legal pornography that is designed specifically to collide with CSAM hashes and result in automated alerting to law enforcement. Pay X monero within the next 30 minutes, or quite literally, you may go to jail, and be charged with possessing child pornography, until you spend $XXX,XXX on lawyers and expert testimony that demonstrates your innocence. * Another delivery mechanism for this is simply sending collided photos over WhatsApp, as WhatsApp allows for up to 30 media images in one message, and has settings that will automatically add these images to your iCloud photo library.
Alupis 5y ago
So, what does Apple get out of all this, except negative attention, erosion of their image, possible privacy lawsuits, etc? I just don't understand what Apple's motivation would have been here. Surely this fallout could have been anticipated?
- Spivak 5y ago
  The FBI off their back that they aren’t doing enough to stop the spread of CP.
  - Alupis 5y ago
    I find it hard to believe CSAM was so pervasive on iDevices that they'd feel compelled to do something about it. As far as we know (and I'm sure lots of eyeballs are looking now) Android doesn't do this. And frankly, why would Apple care that the FBI isn't cozy with them. Their entire brand is "security and privacy", kind of goes against most 3 Letter Agencies anyway.
    - simondotau 5y ago
      I agree CSAM isn't likely to be pervasive in the photo libraries on iOS devices. Android does not do on-device scanning, but Google does scan photos after they are uploaded to their cloud photo service. It's not on-device scanning, but the effect is functionally identical: photos that are being uploaded to the cloud are being scanned for CSAM. The only real distinction is who owns the CPU which computes the hash. I doubt it's the FBI pressuring Apple. My suspicion is it's fear of the US Congress passing worse, even more privacy-invading laws under the guise of combating CSAM. If Apple's lobbyists can show that iPhones are already searching for CSAM, arguments for such laws get weaker.
      - Alupis 5y ago
        > Android doesn't do on-device scanning, but Google does scan photos after they are uploaded to their cloud photo service. So did Apple, and pretty much all cloud hosting providers. This, on device, scanning is what's new, and very out of character for Apple. > If Apple's lobbyists can show that iPhones are already searching for CSAM, arguments for such laws get weaker. I'm not aware of any big anti-CSAM push being made by Congress. CSAM just isn't really a big issue in the US, the existing laws, and culture, are pretty effective already.
vermilingua 5y ago
The integrity of this entire system now relies on the security of the CSAM hash database, which has just dramatically increased in value to potential attackers. All it would take now, is for one CSAM hash to be known to the public, then uploading collided iPhone wallpapers to wallpaper download sites. That many false positives will overload whatever administrative capacity there is to review reports in a matter of days.
- Matheus28 5y ago
  There's no need for someone to get the entire CSAM database. If they go on the darknet and just find enough images (or hashes) that would trip Apple's system, that would be enough. I'd assume any publicly available image on the darknet would likely also be on CSAM.
  - copperx 5y ago
    Exactly. It would be trivial for anyone to compile a list of possible known CSAM hashes. It would be illegal to do so, but only one person has to do it, and then the list of probably positive hashes can be distributed legally around the web.
    - gnulinux 5y ago
      Human doesn't need to see the material. One can automate this by crawling websites with known bad material and compute hash of all images. Surely some of those hashes are CSAM, we don't know which. But it's still equally dangerous.
- simondotau 5y ago
  Before they make it to human review, photos in decrypted vouchers have to pass the CSAM match against a second classifier that Apple keeps to itself.
- ec109685 5y ago
  No, there’s another private hash function that also has to match the known CSAM image for an image to be considered a match. That one can’t be figured out through this technique.
endisneigh 5y ago
Why does matter? The photo looks nothing like the target? If someone looks at the two images wouldn’t they see they’re not the same and therefore the original image was mistakenly linked with the target
- dannyw 5y ago
  Apple's reviewers, by law, cannot look at the target. No one except NCMEC is allowed to possess the target (CSAM material). So Apple will be looking at a low-res grayscale image of whatever the collided image is, which could be legal adult pornography (let's say: a screengrab of legal "teen" 18+ porn), but the CSAM filter tells it that it's abuse material! What would you do as the Apple reviewer? (Hint: You only have one option, as you are legally mandated to report).
  - dragonwriter 5y ago
    > Apple's reviewers, by law, cannot look at the target This is false. > No one except NCMEC is allowed to possess the target (CSAM material). False. No one is allowed to knowingly possess it, without taking certain actions forthwith when they become aware of it. Obviously, prior to it being reviewed as it is, neither the reviewer nor Apple has knowledge that it is actual or even particularly likely CSAM.
    - Dylan16807 5y ago
      I think you misread what was meant by "target". Yeah, Apple might be able to look at the uploaded image. But the reviewers don't have a copy of the original image added to the database, which is the "target".
      - endisneigh 5y ago
        You’re correct, but the uploaded image is sufficient as it would be obvious some that it isn’t CSAM material. If it was, then would it matter if it wasn’t the original?
ipiz0618 5y ago
Great work, I hope people keep hacking the system to lower the system's credibility. This idea is just beyond insane, and the plan to have manual check on user's photos on their own devices sounds like what China is doing - not great
throwaway384950 5y ago
Some people seem to be confused why a hash collision of a cat and a dog matters. Here's a potential attack: share (legal) NSFW pictures that are engineered to have a hash collision with CSAM to get someone else in trouble. The pictures are flagged as CSAM, and they also look suspicious to a human reviewer (maybe not enough context in the image to identify the subject's age). To show that this can be done with real NSFW pictures, here is an example, using an NSFW image from a subreddit's top posts of all time. Here is the image (NSFW!): https://i.ibb.co/Ct64Cnt/nsfw.png https://i.ibb.co/Ct64Cnt/nsfw.png Hash: 59a34eabe31910abfb06f308
- robertoandred 5y ago
  Does anyone save porn to their personal photo libraries? Especially porn as suspicious as the image you posted?
  - throwaway384950 5y ago
    Going by what some people on Reddit say, it seems to be the case. https://old.reddit.com/r/datahoarder/search?q=porn&restrict_sr=on https://old.reddit.com/r/datahoarder/search?q=porn&restrict_... Probably not the weird image I posted, which looks obviously suspicious. But maybe someone will make a program to find "cleaner" hash collisions that don't look suspicious.
    - nullc 5y ago
      https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-902977931 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... I posted some examples that look like totally normal images, they're no harder to produce, you just need to noise-shape the gradient descent so that the introduced noise has a spectrum similar to the image. E.g. just feeding back a gaussian highpassed version of the error signal is sufficient.
  - pseudalopex 5y ago
    The CSAM detection system supposes people save actual child porn to their personal photo libraries.
animanoir 5y ago
Can someone explain me what isn neural hash?
- K5EiS 5y ago
  NeuralHash is a hashing algorithm made by Apple to create hashes from images. Where other hashing algorithms would look at the pixel values, NeuralHash creates hashes based on the visual features of an image. You can read more about it here: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - animanoir 5y ago
    thank you very much
iaw 5y ago
Really naive question. What's to stop apple from using two distinct and separate visual hashing algorithms? Wouldn't the collision likelihood decrease drastically in that scenario? Again, really naive but it seems like if you have two distinct multi-dimensional hashes it would be much harder to solve the gradient descent problem.
- 63 5y ago
  I'm fairly sure they do, actually. It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side after it gets flagged by the client side neural hash. Then only after 30 such images from a single user are identified as CSAM by both algorithms will they be sent to a human reviewer who will confirm their contents. Then, finally, law enforcement will be alerted. There has been a lot of hyperbole going around and the original premise that this is a breach of privacy is still true, but in my opinion the actual repercussions of attacks and collisions are being grossly exaggerated. One would have to create a collision with known CSAM for both algorithms (one of which is secret) which also overlaps with a legal porn image that could be misconstrued as CSAM by a human reviewer, or at the very least create and distribute hundreds of double collisions to DOS the reviewers.
  - silvestrov 5y ago
    > It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side But then they still need to upload the original image to the server, and what was the reason for doing the scanning client-side then when they still upload it?
    - Thorrez 5y ago
      >and what was the reason for doing the scanning client-side then when they still upload it? Probably so that China cannot force Apple to hand over arbitrary images in iCloud (or all images in iCloud). With Apple's design the only images China can get from you are malicious images that people send you. If Apple scanned every image serverside without any clientside scanning, then theoretically China could get all newly-uploaded iCloud images.
blintz 5y ago
I don't see how this is fixable on their end. Several people have suggested simply layering several different perceptual hash systems, with the assumption that it's difficult to find a colliding image in all of them. This is pretty suspect - there's a reason we hold a decades-long competition to select secure hash functions. Basically, a function can't generally achieve cryptographic properties (like collision-resistance, or difficulty of preimage computation) without being specifically designed for it. By it's nature, any perceptual hash function is trivially not collision resistant, and any set of neural models are highly unlikely to be preimage-resistant. The really tough thing to swallow for me is the "It was never supposed to be a cryptographic hash function! It was always going to be easy to make a collision!" line. If this was such an obvious attack, why wasn't it mentioned in any of the 6+ security analyses? Why wasn't it mentioned as a risk in the threat model?
- shuckles 5y ago
  > First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
  - ec109685 5y ago
    And that hash function is kept private, so you can’t just iterate to find a match of both hashes.
    - shuckles 5y ago
      Correct, gradient descent attacks like this depend on being able to differentiate your hash function.
  - blintz 5y ago
SCUSKU 5y ago
Well done! Hopefully all of this progress toward demonstrating how easy it is to manipulate neural hash will get Apple to rollback the update...
- shuckles 5y ago
  Counter-point: hijinks like this are defeated by including the original image instead of the image derivative in associated data. At that point, the system works in the exact same way as the photo scanning status quo.
- jobigoud 5y ago
  Can they though? To the general public the optics of rolling back the update now would be that they are not fighting CSAM.
fnord77 5y ago
how long until they start scanning a device's framebuffer in realtime? why stop at CSAM? Pirated material like movies next?
- heavyset_go 5y ago
  > how long until they start scanning a device's framebuffer in realtime? Some smart TVs do automated content recognition so the manufacturers can spy on what you're watching and sell the data to the highest bidders.
  - Syonyk 5y ago
    Roku's "privacy" policy is a hoot to read for stuff like this. It's basically, "If we've come up with a way to grab it, we do. And send it to our servers. And do what we want with it." It literally includes: > We may receive information about the browser and devices you use to access the Internet, including our services, such as device types and models, unique identifiers (including, for Roku Devices, the Advertising Identifier associated with that device), IP address, operating system type and version, browser type and language, Wi-Fi network name and connection data, and information about other devices connected to the same network. Emphasis mine. They literally have given themselves permission to nmap your LAN and upload the results!
- cirrus3 5y ago
  They are not doing this for the fun of it. If they didn't have to, they would not do it at all. You have made a huge leap from scanning for pre-existing CSAM while in transit to a cloud service to scanning frame buffers on device in real-time. You should get some type of Olympic medal for such a leap. This tech is to catch the lowest possible hanging fruit of the dumbest of all CSAM-sharing/saving folks as required by law.
  - copperx 5y ago
    Real-time framebuffer hashing might be a big leap, but what about local filesystem scanning built into the OS?
deleted 5y ago
[deleted]
spullara 5y ago
Apple is scanning files locally before they are uploaded to iCloud in order to avoid storing unencrypted photos within iCloud but still discovering CSAM. All the other storage providers already scan all the images uploaded on their servers. I guess you can decide which is better. Here is Google's report on it: https://transparencyreport.google.com/child-sexual-abuse-material/reporting?hl=en https://transparencyreport.google.com/child-sexual-abuse-mat...
- jjcon 5y ago
  > in order to avoid storing unencrypted photos within iCloud To be clear, Apple does not utilize E2E in iCloud. They can (and already do) scan iCloud contents
cirrus3 5y ago
What are trying to prove here? It takes a human not noticing that a glitchy image of cat is not the same as picture of dog, 30 times. Yea collisions are technically, possible. Apple has accounted for that. What is your point? Hashes are at the core of a lot of tech, and collisions are way easier and more likely to happen in those in many cases, but suddenly this is an issue for ya'll?
- deleted 5y ago
  [deleted]
ThinBold 5y ago
Despite that Apple scanning our images is a horrible privacy practice, I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Surely you can easily fabricate innocent images whose NeuralHash matches the database. But in what way are you going to send them to victims and convince them to save them to their photo library? The moment you send it via WhatsApp FB will stop you because (they think) it is a problematic image. And Even if the image did land, it has to look like some cats and dogs or the receiver will just ignore. (Even worse, the receiver may report you.) And even if your image does look like cats and dogs, it has to pass another automatic test at the server side that uses another obfuscated, constantly-updating algorithm. After that, even more tests if Apple really wants to. That means your image needs to collide ≥ three times, one open, one obfuscated, and one Turing. Gmail scans your attachments and most people are cool with it. I highly doubt that Apple has any reason to withdraw this.
- istingray 5y ago
  I believe being downvoted because this is thoroughly covered in the thread. Suggest you read it all again.
  - ThinBold 5y ago
    I did my best skimming comments the algorithm showed me, including related posts' comments. But man am I bad at reading comprehension. If I miss anything then surely I am willing to be corrected. But so far I don't see comments that show us how to penetrate the four-layer system (local hash check, semantic check by user, on-server hash check, and human reviewer).
    - istingray 5y ago
      Ah ok! Here's the relevant part of the thread for that. https://news.ycombinator.com/item?id=28229832 https://news.ycombinator.com/item?id=28229832
      - ThinBold 5y ago
yoz-y 5y ago
To me the most interesting findings from this fiasco were: 1. People actually do use generally publicly available services to store and distribute CP (as suggested by the amount of reports done by Facebook) 2. A lot of people evidently use iCloud Photo Library to store images of things other than pictures they took themselves. This is not really surprising, I've learned that the answer of "does anybody ever?" questions is aways "yes". It is a bit weird though since the photo library is terrible for this use case.
- corgihamlet 5y ago
  My iCloud Photo Library is full of pictures saved from the Web and screenshots. That is just the way it works on iOS and it is really annoying to have random cute dog pictures saved from Reddit crop up in "Your Furry Friends"-Compilations.
  - yoz-y 5y ago
    Before Files I've never actually saved any pictures and painstakingly routinely deleted screenshots. The first is now a bit easier since the share sheet also has "save to files" (only some of the time though, for no apparent reason). The second is a bit easier as there at least is a Screenshot automatic album. But yes, I see why people do this, but I wish apple provided a better way to not pollute the iCloud library.
- signal11 5y ago
  > 2. A lot of people evidently use iCloud Photo Library to store images of things other than pictures they took themselves Not in the context of CSAM, but this is iOS’s “appliance” user interface coming back to bite it. The iOS photos app doesn’t appear to have a way to show the user only the photos they took. Apps like Twitter, browsers, chat apps, screenshots all get to save their photos in the photo library. I believe iOS 15 has a way to filter photos by originating app, but for most users currently, it’s hard not to use iCloud Photo Library for photos I didn’t take myself. Interestingly, users save chat history including images, from apps like iMessage and WhatsApp, on iCloud too. I’m not sure what happens to e2e encryption for backed up data.
dang 5y ago
Ongoing related threads: Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706 https://news.ycombinator.com/item?id=28225706 - Aug 2021 (401 comments) Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068 https://news.ycombinator.com/item?id=28219068 - Aug 2021 (662 comments) Convert Apple NeuralHash model for CSAM Detection to ONNX - https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391 - Aug 2021 (177 comments) (I just mean related to this particular project. To list the threads related to larger topic would be...too much.)
deleted 5y ago
[deleted]
guerrilla 5y ago
So when do we start the protest? Everyone could generate plant a bunch ofthese false positives on their devices. If enough people did it, it'd cost them.
romeovs 5y ago
A lot has been said about using this as an attack vector by possibly poisoning a victims iPhone with an image that matches a CSAM hash. But could this not also be used to circumvent the CSAM scanning by converting images that are in the CSAM database to visually similar images that won't match the hash anymore? That would effectively defeat the CSAM scanning Apple and others are trying to put into place completely and render the system moot. One could argue that these spoofed images could also be added to the CSAM database, but what if you spoof them to have hashes of extremely common images (like common memes)? Adding memes to the database would render the whole scheme unmanageable, no? Or am I missing something here? So we'd end up with a system that: 1. Can't be reliably used to track actual criminal offenders (they'd just be able to hide) without rendering the whole database useless. 2. Can be used to attack anyone by making it look like they have criminal content on their iPhones.
- chongli 5y ago
  Or am I missing something here? Wouldn't it be easier for offenders to avoid Apple products? That requires no special computer expertise and involves no risk on their part.
  - romeovs 5y ago
    That would be more effective! But that's not something Apple is trying to solve here (or could ever solve). They are trying to prevent CSAM images from being stored and distributed using Apple products. If that goal is easily circumvented, the whole motivation for this (anti-)feature becomes invalid. I a way, this architecture could potentially even make Apple products more attractive for CSAM distributors, since they now have a known way to fly under the radar (something that is arguably harder/riskier on other image sharing platforms, where the matching happens server-side). One reasonable strategy Apple could have against that is through constantly finetuning the NeuralHash algorithm to hopefully catch more and more offenders. If that works reasonably well, it might deter criminals from their platform because an image that flies under the radar now might not fly under the radar in the future. NB. I'm not trying to say Apple is doing the right thing here, especially since the above arguments put the efficacy of this architecture under scrutiny.
jobigoud 5y ago
Can someone explain what is the profile of criminals they expect to catch with this system? People that are tech savy enough to go on the darknet and find CSAM content but simultaneously stupid enough to upload these images to iCloud? And they think there are enough of these people to create this very complicated system and risk a PR disaster?
- azinman2 5y ago
  Facebook reported 68.1 million csam images last year. If these people were such criminal masterminds, why are Facebook’s numbers so high?
- Erwin 5y ago
  Here in Denmark a 15-year old girl and boy was filmed while having sex, and the video spread around among teenagers, apparently mostly through Facebook Messenger. In 2018, the police indicted 1000 of them (tracking them down with Facebook's help). Legal results were a child-porn law judgement for 334 of them, and simpler penalties for 400. The child-porn judgement was mostly suspended sentences, but it precludes working with children (as a teacher or even sports trainer if children under 15yo are involved) for between 10 and 20 years. If there was a system that would have caught it sooner, prior to sharing, the spread would be minimized. The police took 3 years to form a plan to indict the 1000+ people.
xucheng 5y ago
In addition to the attacks, such as converting legit image to be detected as CSAM (false positive) or circumventing detection of the real CSAM image (false negative), which have been widely discussed in HN, I think this can also be used to mount a DOS attack or to censor any images. It works like this. First, found your target images, which are either widely available like internet memes for DOS attack or images you want to censor. Then, compute their Neuralhash. Next, use the hash collision tool to turn real CSAM images to have the same NeuralHash as the target images. Finally, report these adversarial CSAM images to the government. The result is that the attackers would successfully add the targeted NeuralHash into the CSAM database. And people who store these legit image will then be flagged.
mam3 5y ago
isn't it sufficient that they change the function every day or something ?
saithound 5y ago
I've seen a lot of comments "muddying the waters" (intentionally or not) about whether hash colliders like the one demonstrated above can be used to carry out an attack. So I wrote up a quick FAQ addressing the most common points. Part 1/2 Q: I heard that Apple employees inspect a "visual derivative" of your photos before reporting you to the authorities. Doesn't this mean that, even if you modify images so their hash matches CSAM, the visual derivative won’t match? A: No. "Matching the visual derivative" is completely meaningless. The visual derivative of your photo cannot be matched against anything, and there is no such thing as an "original" visual derivative to match against. Let me elaborate. The visual derivative is nothing more than a low resolution thumbnail of the photo that you uploaded. In this context, a "derivative" simply refers to a transformed, modfied or adapted version of your photo. So a "visual derivative" of your photo means simply a transformed version of your photo that still identifiably looks like the photo you uploaded to iCloud. This thumbnail is never matched against known CSAM thumbnails. The thumbnail cannot be matched against known CSAM thumbnails, most importantly because Apple doesn't possess a database of such thumbnails. Indeed, the whole point of this exercise is that Apple really doesn't want to store CSAM on their servers! Instead, an Apple employee looks at the thumbnails derived from your photos. The only judgment call this employee gets to make is whether it can be ruled out (based on the way the thumbnail looks) that your uploaded photo is CSAM-related. As long as the thumbnail contains a person, or something that looks like the depiction of a person (especially in a vaguely violent or vaguely sexual context, e.g. with nude skin or with injuries) they will not be able to rule out this possibility based on the thumbnail alone. You can try it yourself: consider three perfectly legal and work-safe thumbnails of a famous singer [1]. The singer is underage in precisely one of the three photos. Can you tell which one? All in all, there is no "matching" of the visual derivatives. There is a visual inspection, which means that if you reach a certain threshold, a person will look at thumbnails of your photos. Given the ability to produce hash collisions, an adversary can easily generate photos that fail visual inspection. This can be accomplished straightforwardly by using perfectly legal violent or sexual material to produce the collision (e.g. most people would not suspect foul play if they got a photo of genitals from their Tinder date). But more sophisticated attacks [2] are also possible, especially since the computation of the visual derivative happens on the client, so it can and will be reverse engineered. Q: I heard that there is a second hash function that Apple keeps secret. Isn't it unlikely that an adversarial image would trigger a collision on two distinct hashing algorithms? A: No, it's not unlikely at all. The term "hash function" is a bit of a misnomer. When people hear "hash", they tend to think about cryptographic hash functions, such as SHA256 or BLAKE3. When two messages have the same hash value, we say that they collide. Fortunately, cryptographic hash functions have several good properties associated with them: for example, there is no known way to generate a message that yields a given predetermined hash value, no known way to find two different messages with the same hash value, and no known way to make a small change to a message without changing the corresponding hash value. These properties make cryptographic hash functions secure, trustworthy and collision-resistant even in the face of powerful adversaries. Generally, when you decide to use two unrelated cryptographic hash algorithms instead of one, you make finding a collision at least twice as difficult for the adversary. However, the hash functions that Apple uses for identifying CSAM images are not "cryptographic hash functions" at all. They are "perceptual hash functions". The purpose of a perceptual hash is the exact opposite of a cryptographic hash: two images that humans see/hear/perceive (hence the term perceptual) to be the same or similar should have the same perceptual hash. There is no known perceptual hash function that remains secure and trustworthy in any sense in the face of (even unsophisticated) adversaries. Most importantly, it is not guaranteed that using two unrelated perceptual hash functions makes finding collisions more difficult. In fact, in many contexts, these adversarial attacks tend to transfer: if they work against one model, they often work against other models as well [3]. To make matters worse, a second, secret hash function can be used only after the collision threshold has been passed (otherwise, it would have to be done on the device, but then it cannot be kept secret). Since the safety voucher is not linked directly to a full resolution photo, the second hashing has to be performed on the tiny "visual derivative", which makes collisions all the more likely. Apple's second hash algorithm is kept secret (so much so that the whitepapers released by Apple do not claim and do not confirm its existence!). This means that we don't know how well it works. We can't even rule out the second hash algorithm being a trivial variation (or completely identical) to the first hash algorithm. Moreover, it's unlikely that the second algorithm was trained on a completely different dataset than the first one (e.g. because there are not many such hash algorithms that work well; moreover, the database of known CSAM content is really quite small compared to the large datasets that good machine learning algorithms require, so testing is necessarily limited). This suggests that transfer attacks are likely to work.
m3kw9 5y ago
So is this going to be used for DDOSing their photo verifying service?

623 points by anishathalye 5y ago

The README (https://github.com/anishathalye/neural-hash-collider#how-it-works https://github.com/anishathalye/neural-hash-collider#how-it-...) explains in a bit more detail how these adversarial attacks work. This code pretty much implements a standard adversarial attack against NeuralHash. One slightly interesting part was replacing the thresholding with a differentiable approximation. I figured I'd share this here in case anyone is interested in seeing what the code to generate adversarial examples looks like; I don't think anyone in the big thread on the topic (https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...) has shared attack code for NeuralHash in particular yet.

IncRnd 5y ago
Nicely done. Thank you for sharing. I'd like to share the following paper for anyone else who may be interested. It is about watermarking rather than a preimage attack. "Adversarial Embedding: A robust and elusive Steganography and Watermarking technique" https://arxiv.org/abs/1912.01487 https://arxiv.org/abs/1912.01487 Unfortunately, the existence of invisible watermarking demonstrates a separate attack on the hash. Instead of a preimage attack, this might be able to change the hash of an image that is suspected of already being a match. A true-positive would be changed into a false-negative.
GistNoesis 5y ago
Well done ! Here is my version that uses Scipy lbfgs-b optimizers : https://gist.github.com/unrealwill/d64d653a7626b825ef332aa3b2aa1a43 https://gist.github.com/unrealwill/d64d653a7626b825ef332aa3b...

Banditoz 5y ago

That was scary fast. Is there a point in using this algorithm for its intended purpose now?

dannyw 5y ago
If the intended purpose is to lead by example and eventually mandate code on every computing device (phone and computer) that scans all files against a government provided database, then yes, that purpose still exists and this algorithm still works for it. Just wait and watch - I guarantee you that Apple will be talking about CSAM in at least one anti-trust legal battle about why they shouldn't be broken up. Because a walled garden means they can oppress citizens on behalf of governments better.
zepto 5y ago
Yes, because isn’t a weakness in the design. There is nothing scary fast about it. It was obvious and anticipated in the threat model.

seph-reed 5y ago

Is it possible to host this online as a meme filter? I think every meme should get pumped through this, just for lulz.

m2com 5y ago

Hi, this looks interesting but I have no idea what this all means lol Is this some way of hiding a picture within a picture, or am I way off the mark?

sodality2 5y ago
Apple began scanning for CSAM with Neuralhash. This allows you to turn an image into a specific neuralhash thus possibly triggering its (automatic) CSAM detection. Imagine if a picture of a cat could cause Apple to think you have CP on your device.
- m2com 5y ago
  Oh damn, that’s crazy. Very cool project. Thanks for sharing and the explanation.
- neom 5y ago
  Well, you'd have to do it 30 times to trigger the system, and then someone at apple moderation would look at those 30 pictures of cats and hit "next" vs "supervisor"
  - m2com 5y ago
    Good that there’s some human supervision. But, I know I have more than 30 photos of my dog. Also don’t like the idea of false positives auto-sharing some of my camera roll.
    - neom 5y ago
      It's only if you back it up to iCloud, the signatures of the CP used as references are rotated, and they're also not public. The chances of you randomly triggering the system is effectively 0 unless you're uploading CP to your iCloud.
      - Alupis 5y ago
        Wait, wasn't all the hullabaloo over this scanning not requiring an upload to iCloud anymore? They're scanning anything you upload to iCloud (and have been for some time) but now also scan everything on your device too.

ryanmarsh 5y ago

Ok so now all we have to do is get a phone, load it with adversarial images that have hashes from the CSAM database and we wait and see what happens. Basically a honeypot. Get some top civil rights attorneys involved. Take the case to the Supreme Court. Get precedence set right. Lawfare

robertoandred 5y ago
Where would you get the CSAM hashes?
- dannyw 5y ago
  Give it a few days, and you'll probably find someone selling a list of CSAM neural hashes on darknet marketplaces.
  - Dylan16807 5y ago
    Or tweeting out a bunch of them. They're just 12 byte numbers.
    - copperx 5y ago
      I bet there's a list of hashes already up in Pastebin.
- arsome 5y ago
  The client has to be able to check for them in some way - just run that algorithm against every image you can scrape from Tor/Freenet and I suspect you'll have results rather quickly. Or you can probably just wait a minute and pay an... enterprising individual to sell you such a list on a darknet market though, or perhaps even find one posted on the clearnet soon enough.
  - robertoandred 5y ago
    No, the client doesn’t have access to the CSAM hashes. And matches are verified on the server, not on the client.

deleted 5y ago

[deleted]

ncw96 5y ago

Apple has said this is not the final version of the hashing algorithm they will be using: https://www.vice.com/en/article/wx5yzq/apple-defends-its-anti-child-abuse-imagery-tech-after-claims-of-hash-collisions https://www.vice.com/en/article/wx5yzq/apple-defends-its-ant...

only_as_i_fall 5y ago
Does it matter? Unless they're going to totally change the technology I don't see how they can do anything but buy time until it's reverse engineered. After all, the code runs locally. If Apple wants to defend this they should try to explain how the system will work even if generating adversarial images is trivial.
- ncw96 5y ago
  Apple has outlined[1] multiple levels of protection in place for this: 1. You have to reach a threshold of matches before your account is flagged. 2. Once the threshold is reached, the matched images are checked against a different perceptual hash algorithm on Apple servers. This means an adversarial image would have to trigger a collision on two distinct hashing algorithms. 3. If both hash algorithms show a match, then “visual derivative” (low-res versions) of the images are inspected by Apple to confirm they are CSAM. Only after these three criteria are met is your account disabled and referred to NCMEC. NCMEC will then do their own review of the flagged images and refer to law enforcement if necessary. [1]: https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
  - Dylan16807 5y ago
    I do want to note that decrypting the low-res images would have to happen before step 2.
  - copperx 5y ago
    Will the high-resolution images be collected and used as evidence? Or just the visual derivatives? That's not clear.

jchw 5y ago

I am strongly against Apple’s decision to do on-device CSAM detection, but: wasn’t there a secondary hash whose database is not shared? In theory you need to collide with both to truly defeat the design, right?

PBnFlash 5y ago
You just need a sample of something that is evidently so pervasive we're building a nation wide dragnet to stop.
- jchw 5y ago
  I doubt finding it would really be that hard, if you wanted to be on a list somewhere in some government database, but even armed with a full image that is in the NCMEC database, the problem is that the second hashing algorithm runs on the server and presumably has secure-by-obscurity details… so it would be hard to collide with it on purpose unless you are an insider. That’s my understanding, although details have been a bit shaky at times.
only_as_i_fall 5y ago
I don't see how. They're hashing on feature space (so trivial cropping and such doesn't defeat this) but they have two totally separate methods of matching those hashes? Doesn't sound right to me...
- jchw 5y ago
  Apparently the images in question would get sent to the server, and all calculation happens there. > In a call with reporters regarding the new findings, Apple said its CSAM-scanning system had been built with collisions in mind, given the known limitations of perceptual hashing algorithms. In particular, the company emphasized a secondary server-side hashing algorithm, separate from NeuralHash, the specifics of which are not public. If an image that produced a NeuralHash collision were flagged by the system, it would be checked against the secondary system and identified as an error before reaching human moderators. https://www.theverge.com/2021/8/18/22630439/apple-csam-neuralhash-collision-vulnerability-flaw-cryptography https://www.theverge.com/2021/8/18/22630439/apple-csam-neura... For one reason or another Apple really wants to create this precedent, so it’s only natural they’re doing every last thing to make the feature hard to defeat.

robertoandred 5y ago

…and? Does OP think reviewers will think a picture of a cat is CSAM?

tibbar 5y ago
The input image is a parameter, so you could start with some image that would be easier to confuse at the low resolutions the reviewers use. The missing piece in this case is that the database of target hashes is unknown.
- Dylan16807 5y ago
  Easier to confuse, or even something like a pic of a 20 year old where it's effectively impossible to be sure from the image itself.

Scaevolus 5y ago

NeuralHash collisions are interesting, but the way Apple is implementing their scanner it's impossible to extract the banned hashes directly from the local database. There are other ways to guess what the hashes are, but I can't think of legal ones. > Matching-Database Setup. The system begins by setting up the matching database using the known CSAM image hashes provided by NCMEC and other child-safety organizations. First, Apple receives the NeuralHashes corresponding to known CSAM from the above child-safety organizations. Next, these NeuralHashes go through a series of transformations that includes a final blinding step, powered by elliptic curve cryptography. The blinding is done using a server-side blinding secret, known only to Apple. The blinded CSAM hashes are placed in a hash table, where the position in the hash table is purely a function of the NeuralHash of the CSAM image. This blinded database is securely stored on users’ devices. The properties of elliptic curve cryptography ensure that no device can infer anything about the underlying CSAM image hashes from the blinded database. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

dannyw 5y ago
You can extract the hashes with a few hours spent on the darknet. Doing that is certainly illegal and not to mention VERY morally wrong, but criminals exist and criminals won't hesitate to abuse this as a mechanism for framing, extortion, or ransom. It's also possible for someone (Attacker A) to go on the darknet and get a list of 96-bit neural hashes, and then publish or sell this list somewhere to another party, Attacker B. The second party would never have to interact with CSAM. Imagine Ransomware v2: We have inserted 29 photos of CSAM-matching material into your photo library. Pay X monero to this address in 30 minutes, or we will insert 2 additional photos, which will cross the threshold and may result in serious and life-changing consequences to you[1]. The difference here (versus the status quo) is that an easily-broken perceptual hashing enables the attacker to never send or possess any CSAM images[2]. From my experiences with being victims of various hackers, I know a lot of them won't touch CSAM because they know it's wrong, but they'll salivate at an opportunity to weaponise automated CSAM scanning. [1]: If you think Apple's human review will mitigate this attack, you can permute legal pornography to match CSAM signatures. If Apple's reviewers see 30 CSAM matches and the visual derivatives look like porn, they will be legally required to report to to NCMEC (a statutory quasi-government agency staffed by the FBI), even if all the photos are actually consensual adults. [2]: If you never possess nor touch CSAM, it might be harder for you to get charged with CP charges. You might be looking at CFAA, blackmail or extortion charges; while your victim faces child pornography charges. This is basically an "amplification attack" on the real world judicial system.

onepunchedman 5y ago

This is just getting wilder and wilder by the day, how spectacularly this move has backfired. As others have commented, at this point all you need is someone willing to sell you the CSAM hashes on the darknet, and this system is transparently broken. Until that day, just send known CSAM to any person you'd like to get in trouble (make sure they have icloud sync enabled), be it your neighbour or a political figure, and start a PR campaign accusing the person of being investigated for it. The whole concept is so inherently flawed it's crazy they haven't been sued yet.

deleted 5y ago
[deleted]
dannyw 5y ago
The "send known CSAM" attack has existed for a while but never made sense. However, this technology enables a new class of attacks: "send legal porn, collided to match CSAM perceptual hashes". With the previous status quo: 1. The attacker faces charges of possessing and distributing child pornography 2. The victim may be investigated and charged with child pornography if LEO is somehow alerted (which requires work, and can be traced to the attacker). Poor risk/reward payoff, specifically the risk outweighs the reward. So it doesn't happen (often). --- With the new status quo of lossy, on-device CSAM scanning and automated LEO alerting: 1. The attacker never sends CSAM, only material that collides with CSAM hashes. They will be looking at charges of CFAA, extortion, and blackmail. 2. The victim will be automatically investigated by law enforcement, due to Apple's "Safety Voucher" system. The victim will be investigated for possessing child pornography, particularly if the attacker collides legal pornography that may fool a reviewer inspecting a 'visual derivative'. Great risk/reward payoff. The reward dramatically outweighs the risk, as you can get someone in trouble for CSAM without ever touching CSAM yourself. If you think ransomware is bad, just imagine CSAM-collision ransomware. Your files will be replaced* with legal pornography that is designed specifically to collide with CSAM hashes and result in automated alerting to law enforcement. Pay X monero within the next 30 minutes, or quite literally, you may go to jail, and be charged with possessing child pornography, until you spend $XXX,XXX on lawyers and expert testimony that demonstrates your innocence. * Another delivery mechanism for this is simply sending collided photos over WhatsApp, as WhatsApp allows for up to 30 media images in one message, and has settings that will automatically add these images to your iCloud photo library.

Alupis 5y ago

So, what does Apple get out of all this, except negative attention, erosion of their image, possible privacy lawsuits, etc? I just don't understand what Apple's motivation would have been here. Surely this fallout could have been anticipated?

Spivak 5y ago
The FBI off their back that they aren’t doing enough to stop the spread of CP.
- Alupis 5y ago
  I find it hard to believe CSAM was so pervasive on iDevices that they'd feel compelled to do something about it. As far as we know (and I'm sure lots of eyeballs are looking now) Android doesn't do this. And frankly, why would Apple care that the FBI isn't cozy with them. Their entire brand is "security and privacy", kind of goes against most 3 Letter Agencies anyway.
  - simondotau 5y ago
    I agree CSAM isn't likely to be pervasive in the photo libraries on iOS devices. Android does not do on-device scanning, but Google does scan photos after they are uploaded to their cloud photo service. It's not on-device scanning, but the effect is functionally identical: photos that are being uploaded to the cloud are being scanned for CSAM. The only real distinction is who owns the CPU which computes the hash. I doubt it's the FBI pressuring Apple. My suspicion is it's fear of the US Congress passing worse, even more privacy-invading laws under the guise of combating CSAM. If Apple's lobbyists can show that iPhones are already searching for CSAM, arguments for such laws get weaker.
    - Alupis 5y ago
      > Android doesn't do on-device scanning, but Google does scan photos after they are uploaded to their cloud photo service. So did Apple, and pretty much all cloud hosting providers. This, on device, scanning is what's new, and very out of character for Apple. > If Apple's lobbyists can show that iPhones are already searching for CSAM, arguments for such laws get weaker. I'm not aware of any big anti-CSAM push being made by Congress. CSAM just isn't really a big issue in the US, the existing laws, and culture, are pretty effective already.

vermilingua 5y ago

The integrity of this entire system now relies on the security of the CSAM hash database, which has just dramatically increased in value to potential attackers. All it would take now, is for one CSAM hash to be known to the public, then uploading collided iPhone wallpapers to wallpaper download sites. That many false positives will overload whatever administrative capacity there is to review reports in a matter of days.

Matheus28 5y ago
There's no need for someone to get the entire CSAM database. If they go on the darknet and just find enough images (or hashes) that would trip Apple's system, that would be enough. I'd assume any publicly available image on the darknet would likely also be on CSAM.
- copperx 5y ago
  Exactly. It would be trivial for anyone to compile a list of possible known CSAM hashes. It would be illegal to do so, but only one person has to do it, and then the list of probably positive hashes can be distributed legally around the web.
  - gnulinux 5y ago
    Human doesn't need to see the material. One can automate this by crawling websites with known bad material and compute hash of all images. Surely some of those hashes are CSAM, we don't know which. But it's still equally dangerous.
simondotau 5y ago
Before they make it to human review, photos in decrypted vouchers have to pass the CSAM match against a second classifier that Apple keeps to itself.
ec109685 5y ago
No, there’s another private hash function that also has to match the known CSAM image for an image to be considered a match. That one can’t be figured out through this technique.

endisneigh 5y ago

Why does matter? The photo looks nothing like the target? If someone looks at the two images wouldn’t they see they’re not the same and therefore the original image was mistakenly linked with the target

dannyw 5y ago
Apple's reviewers, by law, cannot look at the target. No one except NCMEC is allowed to possess the target (CSAM material). So Apple will be looking at a low-res grayscale image of whatever the collided image is, which could be legal adult pornography (let's say: a screengrab of legal "teen" 18+ porn), but the CSAM filter tells it that it's abuse material! What would you do as the Apple reviewer? (Hint: You only have one option, as you are legally mandated to report).
- dragonwriter 5y ago
  > Apple's reviewers, by law, cannot look at the target This is false. > No one except NCMEC is allowed to possess the target (CSAM material). False. No one is allowed to knowingly possess it, without taking certain actions forthwith when they become aware of it. Obviously, prior to it being reviewed as it is, neither the reviewer nor Apple has knowledge that it is actual or even particularly likely CSAM.
  - Dylan16807 5y ago
    I think you misread what was meant by "target". Yeah, Apple might be able to look at the uploaded image. But the reviewers don't have a copy of the original image added to the database, which is the "target".
    - endisneigh 5y ago
      You’re correct, but the uploaded image is sufficient as it would be obvious some that it isn’t CSAM material. If it was, then would it matter if it wasn’t the original?

ipiz0618 5y ago

Great work, I hope people keep hacking the system to lower the system's credibility. This idea is just beyond insane, and the plan to have manual check on user's photos on their own devices sounds like what China is doing - not great

throwaway384950 5y ago

Some people seem to be confused why a hash collision of a cat and a dog matters. Here's a potential attack: share (legal) NSFW pictures that are engineered to have a hash collision with CSAM to get someone else in trouble. The pictures are flagged as CSAM, and they also look suspicious to a human reviewer (maybe not enough context in the image to identify the subject's age). To show that this can be done with real NSFW pictures, here is an example, using an NSFW image from a subreddit's top posts of all time. Here is the image (NSFW!): https://i.ibb.co/Ct64Cnt/nsfw.png https://i.ibb.co/Ct64Cnt/nsfw.png Hash: 59a34eabe31910abfb06f308

robertoandred 5y ago
Does anyone save porn to their personal photo libraries? Especially porn as suspicious as the image you posted?
- throwaway384950 5y ago
  Going by what some people on Reddit say, it seems to be the case. https://old.reddit.com/r/datahoarder/search?q=porn&restrict_sr=on https://old.reddit.com/r/datahoarder/search?q=porn&restrict_... Probably not the weird image I posted, which looks obviously suspicious. But maybe someone will make a program to find "cleaner" hash collisions that don't look suspicious.
  - nullc 5y ago
    https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-902977931 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... I posted some examples that look like totally normal images, they're no harder to produce, you just need to noise-shape the gradient descent so that the introduced noise has a spectrum similar to the image. E.g. just feeding back a gaussian highpassed version of the error signal is sufficient.
- pseudalopex 5y ago
  The CSAM detection system supposes people save actual child porn to their personal photo libraries.

animanoir 5y ago

Can someone explain me what isn neural hash?

K5EiS 5y ago
NeuralHash is a hashing algorithm made by Apple to create hashes from images. Where other hashing algorithms would look at the pixel values, NeuralHash creates hashes based on the visual features of an image. You can read more about it here: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- animanoir 5y ago
  thank you very much

iaw 5y ago

Really naive question. What's to stop apple from using two distinct and separate visual hashing algorithms? Wouldn't the collision likelihood decrease drastically in that scenario? Again, really naive but it seems like if you have two distinct multi-dimensional hashes it would be much harder to solve the gradient descent problem.

63 5y ago
I'm fairly sure they do, actually. It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side after it gets flagged by the client side neural hash. Then only after 30 such images from a single user are identified as CSAM by both algorithms will they be sent to a human reviewer who will confirm their contents. Then, finally, law enforcement will be alerted. There has been a lot of hyperbole going around and the original premise that this is a breach of privacy is still true, but in my opinion the actual repercussions of attacks and collisions are being grossly exaggerated. One would have to create a collision with known CSAM for both algorithms (one of which is secret) which also overlaps with a legal porn image that could be misconstrued as CSAM by a human reviewer, or at the very least create and distribute hundreds of double collisions to DOS the reviewers.
- silvestrov 5y ago
  > It was in one of the articles earlier today that Apple has a distinct, secret algorithm they perform on suspected CSAM server side But then they still need to upload the original image to the server, and what was the reason for doing the scanning client-side then when they still upload it?
  - Thorrez 5y ago
    >and what was the reason for doing the scanning client-side then when they still upload it? Probably so that China cannot force Apple to hand over arbitrary images in iCloud (or all images in iCloud). With Apple's design the only images China can get from you are malicious images that people send you. If Apple scanned every image serverside without any clientside scanning, then theoretically China could get all newly-uploaded iCloud images.

blintz 5y ago

I don't see how this is fixable on their end. Several people have suggested simply layering several different perceptual hash systems, with the assumption that it's difficult to find a colliding image in all of them. This is pretty suspect - there's a reason we hold a decades-long competition to select secure hash functions. Basically, a function can't generally achieve cryptographic properties (like collision-resistance, or difficulty of preimage computation) without being specifically designed for it. By it's nature, any perceptual hash function is trivially not collision resistant, and any set of neural models are highly unlikely to be preimage-resistant. The really tough thing to swallow for me is the "It was never supposed to be a cryptographic hash function! It was always going to be easy to make a collision!" line. If this was such an obvious attack, why wasn't it mentioned in any of the 6+ security analyses? Why wasn't it mentioned as a risk in the threat model?

shuckles 5y ago
> First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
- ec109685 5y ago
  And that hash function is kept private, so you can’t just iterate to find a match of both hashes.
  - shuckles 5y ago
    Correct, gradient descent attacks like this depend on being able to differentiate your hash function.
- blintz 5y ago

SCUSKU 5y ago

Well done! Hopefully all of this progress toward demonstrating how easy it is to manipulate neural hash will get Apple to rollback the update...

shuckles 5y ago
Counter-point: hijinks like this are defeated by including the original image instead of the image derivative in associated data. At that point, the system works in the exact same way as the photo scanning status quo.
jobigoud 5y ago
Can they though? To the general public the optics of rolling back the update now would be that they are not fighting CSAM.

fnord77 5y ago

how long until they start scanning a device's framebuffer in realtime? why stop at CSAM? Pirated material like movies next?

heavyset_go 5y ago
> how long until they start scanning a device's framebuffer in realtime? Some smart TVs do automated content recognition so the manufacturers can spy on what you're watching and sell the data to the highest bidders.
- Syonyk 5y ago
  Roku's "privacy" policy is a hoot to read for stuff like this. It's basically, "If we've come up with a way to grab it, we do. And send it to our servers. And do what we want with it." It literally includes: > We may receive information about the browser and devices you use to access the Internet, including our services, such as device types and models, unique identifiers (including, for Roku Devices, the Advertising Identifier associated with that device), IP address, operating system type and version, browser type and language, Wi-Fi network name and connection data, and information about other devices connected to the same network. Emphasis mine. They literally have given themselves permission to nmap your LAN and upload the results!
cirrus3 5y ago
They are not doing this for the fun of it. If they didn't have to, they would not do it at all. You have made a huge leap from scanning for pre-existing CSAM while in transit to a cloud service to scanning frame buffers on device in real-time. You should get some type of Olympic medal for such a leap. This tech is to catch the lowest possible hanging fruit of the dumbest of all CSAM-sharing/saving folks as required by law.
- copperx 5y ago
  Real-time framebuffer hashing might be a big leap, but what about local filesystem scanning built into the OS?

deleted 5y ago

[deleted]

spullara 5y ago

Apple is scanning files locally before they are uploaded to iCloud in order to avoid storing unencrypted photos within iCloud but still discovering CSAM. All the other storage providers already scan all the images uploaded on their servers. I guess you can decide which is better. Here is Google's report on it: https://transparencyreport.google.com/child-sexual-abuse-material/reporting?hl=en https://transparencyreport.google.com/child-sexual-abuse-mat...

jjcon 5y ago
> in order to avoid storing unencrypted photos within iCloud To be clear, Apple does not utilize E2E in iCloud. They can (and already do) scan iCloud contents

cirrus3 5y ago

What are trying to prove here? It takes a human not noticing that a glitchy image of cat is not the same as picture of dog, 30 times. Yea collisions are technically, possible. Apple has accounted for that. What is your point? Hashes are at the core of a lot of tech, and collisions are way easier and more likely to happen in those in many cases, but suddenly this is an issue for ya'll?

deleted 5y ago
[deleted]

ThinBold 5y ago

Despite that Apple scanning our images is a horrible privacy practice, I don't get why 𝚜̶𝚘̶ ̶𝚖̶𝚊̶𝚗̶𝚢̶ some people think this is an ineffective idea. Surely you can easily fabricate innocent images whose NeuralHash matches the database. But in what way are you going to send them to victims and convince them to save them to their photo library? The moment you send it via WhatsApp FB will stop you because (they think) it is a problematic image. And Even if the image did land, it has to look like some cats and dogs or the receiver will just ignore. (Even worse, the receiver may report you.) And even if your image does look like cats and dogs, it has to pass another automatic test at the server side that uses another obfuscated, constantly-updating algorithm. After that, even more tests if Apple really wants to. That means your image needs to collide ≥ three times, one open, one obfuscated, and one Turing. Gmail scans your attachments and most people are cool with it. I highly doubt that Apple has any reason to withdraw this.

istingray 5y ago
I believe being downvoted because this is thoroughly covered in the thread. Suggest you read it all again.
- ThinBold 5y ago
  I did my best skimming comments the algorithm showed me, including related posts' comments. But man am I bad at reading comprehension. If I miss anything then surely I am willing to be corrected. But so far I don't see comments that show us how to penetrate the four-layer system (local hash check, semantic check by user, on-server hash check, and human reviewer).
  - istingray 5y ago
    Ah ok! Here's the relevant part of the thread for that. https://news.ycombinator.com/item?id=28229832 https://news.ycombinator.com/item?id=28229832
    - ThinBold 5y ago

yoz-y 5y ago

To me the most interesting findings from this fiasco were: 1. People actually do use generally publicly available services to store and distribute CP (as suggested by the amount of reports done by Facebook) 2. A lot of people evidently use iCloud Photo Library to store images of things other than pictures they took themselves. This is not really surprising, I've learned that the answer of "does anybody ever?" questions is aways "yes". It is a bit weird though since the photo library is terrible for this use case.

corgihamlet 5y ago
My iCloud Photo Library is full of pictures saved from the Web and screenshots. That is just the way it works on iOS and it is really annoying to have random cute dog pictures saved from Reddit crop up in "Your Furry Friends"-Compilations.
- yoz-y 5y ago
  Before Files I've never actually saved any pictures and painstakingly routinely deleted screenshots. The first is now a bit easier since the share sheet also has "save to files" (only some of the time though, for no apparent reason). The second is a bit easier as there at least is a Screenshot automatic album. But yes, I see why people do this, but I wish apple provided a better way to not pollute the iCloud library.
signal11 5y ago
> 2. A lot of people evidently use iCloud Photo Library to store images of things other than pictures they took themselves Not in the context of CSAM, but this is iOS’s “appliance” user interface coming back to bite it. The iOS photos app doesn’t appear to have a way to show the user only the photos they took. Apps like Twitter, browsers, chat apps, screenshots all get to save their photos in the photo library. I believe iOS 15 has a way to filter photos by originating app, but for most users currently, it’s hard not to use iCloud Photo Library for photos I didn’t take myself. Interestingly, users save chat history including images, from apps like iMessage and WhatsApp, on iCloud too. I’m not sure what happens to e2e encryption for backed up data.

dang 5y ago

Ongoing related threads: Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’ - https://news.ycombinator.com/item?id=28225706 https://news.ycombinator.com/item?id=28225706 - Aug 2021 (401 comments) Hash collision in Apple NeuralHash model - https://news.ycombinator.com/item?id=28219068 https://news.ycombinator.com/item?id=28219068 - Aug 2021 (662 comments) Convert Apple NeuralHash model for CSAM Detection to ONNX - https://news.ycombinator.com/item?id=28218391 https://news.ycombinator.com/item?id=28218391 - Aug 2021 (177 comments) (I just mean related to this particular project. To list the threads related to larger topic would be...too much.)

deleted 5y ago

[deleted]

guerrilla 5y ago

So when do we start the protest? Everyone could generate plant a bunch ofthese false positives on their devices. If enough people did it, it'd cost them.

romeovs 5y ago

A lot has been said about using this as an attack vector by possibly poisoning a victims iPhone with an image that matches a CSAM hash. But could this not also be used to circumvent the CSAM scanning by converting images that are in the CSAM database to visually similar images that won't match the hash anymore? That would effectively defeat the CSAM scanning Apple and others are trying to put into place completely and render the system moot. One could argue that these spoofed images could also be added to the CSAM database, but what if you spoof them to have hashes of extremely common images (like common memes)? Adding memes to the database would render the whole scheme unmanageable, no? Or am I missing something here? So we'd end up with a system that: 1. Can't be reliably used to track actual criminal offenders (they'd just be able to hide) without rendering the whole database useless. 2. Can be used to attack anyone by making it look like they have criminal content on their iPhones.

chongli 5y ago
Or am I missing something here? Wouldn't it be easier for offenders to avoid Apple products? That requires no special computer expertise and involves no risk on their part.
- romeovs 5y ago
  That would be more effective! But that's not something Apple is trying to solve here (or could ever solve). They are trying to prevent CSAM images from being stored and distributed using Apple products. If that goal is easily circumvented, the whole motivation for this (anti-)feature becomes invalid. I a way, this architecture could potentially even make Apple products more attractive for CSAM distributors, since they now have a known way to fly under the radar (something that is arguably harder/riskier on other image sharing platforms, where the matching happens server-side). One reasonable strategy Apple could have against that is through constantly finetuning the NeuralHash algorithm to hopefully catch more and more offenders. If that works reasonably well, it might deter criminals from their platform because an image that flies under the radar now might not fly under the radar in the future. NB. I'm not trying to say Apple is doing the right thing here, especially since the above arguments put the efficacy of this architecture under scrutiny.

jobigoud 5y ago

Can someone explain what is the profile of criminals they expect to catch with this system? People that are tech savy enough to go on the darknet and find CSAM content but simultaneously stupid enough to upload these images to iCloud? And they think there are enough of these people to create this very complicated system and risk a PR disaster?

azinman2 5y ago
Facebook reported 68.1 million csam images last year. If these people were such criminal masterminds, why are Facebook’s numbers so high?
Erwin 5y ago
Here in Denmark a 15-year old girl and boy was filmed while having sex, and the video spread around among teenagers, apparently mostly through Facebook Messenger. In 2018, the police indicted 1000 of them (tracking them down with Facebook's help). Legal results were a child-porn law judgement for 334 of them, and simpler penalties for 400. The child-porn judgement was mostly suspended sentences, but it precludes working with children (as a teacher or even sports trainer if children under 15yo are involved) for between 10 and 20 years. If there was a system that would have caught it sooner, prior to sharing, the spread would be minimized. The police took 3 years to form a plan to indict the 1000+ people.

xucheng 5y ago

In addition to the attacks, such as converting legit image to be detected as CSAM (false positive) or circumventing detection of the real CSAM image (false negative), which have been widely discussed in HN, I think this can also be used to mount a DOS attack or to censor any images. It works like this. First, found your target images, which are either widely available like internet memes for DOS attack or images you want to censor. Then, compute their Neuralhash. Next, use the hash collision tool to turn real CSAM images to have the same NeuralHash as the target images. Finally, report these adversarial CSAM images to the government. The result is that the attackers would successfully add the targeted NeuralHash into the CSAM database. And people who store these legit image will then be flagged.

mam3 5y ago

isn't it sufficient that they change the function every day or something ?

saithound 5y ago

I've seen a lot of comments "muddying the waters" (intentionally or not) about whether hash colliders like the one demonstrated above can be used to carry out an attack. So I wrote up a quick FAQ addressing the most common points. Part 1/2 Q: I heard that Apple employees inspect a "visual derivative" of your photos before reporting you to the authorities. Doesn't this mean that, even if you modify images so their hash matches CSAM, the visual derivative won’t match? A: No. "Matching the visual derivative" is completely meaningless. The visual derivative of your photo cannot be matched against anything, and there is no such thing as an "original" visual derivative to match against. Let me elaborate. The visual derivative is nothing more than a low resolution thumbnail of the photo that you uploaded. In this context, a "derivative" simply refers to a transformed, modfied or adapted version of your photo. So a "visual derivative" of your photo means simply a transformed version of your photo that still identifiably looks like the photo you uploaded to iCloud. This thumbnail is never matched against known CSAM thumbnails. The thumbnail cannot be matched against known CSAM thumbnails, most importantly because Apple doesn't possess a database of such thumbnails. Indeed, the whole point of this exercise is that Apple really doesn't want to store CSAM on their servers! Instead, an Apple employee looks at the thumbnails derived from your photos. The only judgment call this employee gets to make is whether it can be ruled out (based on the way the thumbnail looks) that your uploaded photo is CSAM-related. As long as the thumbnail contains a person, or something that looks like the depiction of a person (especially in a vaguely violent or vaguely sexual context, e.g. with nude skin or with injuries) they will not be able to rule out this possibility based on the thumbnail alone. You can try it yourself: consider three perfectly legal and work-safe thumbnails of a famous singer [1]. The singer is underage in precisely one of the three photos. Can you tell which one? All in all, there is no "matching" of the visual derivatives. There is a visual inspection, which means that if you reach a certain threshold, a person will look at thumbnails of your photos. Given the ability to produce hash collisions, an adversary can easily generate photos that fail visual inspection. This can be accomplished straightforwardly by using perfectly legal violent or sexual material to produce the collision (e.g. most people would not suspect foul play if they got a photo of genitals from their Tinder date). But more sophisticated attacks [2] are also possible, especially since the computation of the visual derivative happens on the client, so it can and will be reverse engineered. Q: I heard that there is a second hash function that Apple keeps secret. Isn't it unlikely that an adversarial image would trigger a collision on two distinct hashing algorithms? A: No, it's not unlikely at all. The term "hash function" is a bit of a misnomer. When people hear "hash", they tend to think about cryptographic hash functions, such as SHA256 or BLAKE3. When two messages have the same hash value, we say that they collide. Fortunately, cryptographic hash functions have several good properties associated with them: for example, there is no known way to generate a message that yields a given predetermined hash value, no known way to find two different messages with the same hash value, and no known way to make a small change to a message without changing the corresponding hash value. These properties make cryptographic hash functions secure, trustworthy and collision-resistant even in the face of powerful adversaries. Generally, when you decide to use two unrelated cryptographic hash algorithms instead of one, you make finding a collision at least twice as difficult for the adversary. However, the hash functions that Apple uses for identifying CSAM images are not "cryptographic hash functions" at all. They are "perceptual hash functions". The purpose of a perceptual hash is the exact opposite of a cryptographic hash: two images that humans see/hear/perceive (hence the term perceptual) to be the same or similar should have the same perceptual hash. There is no known perceptual hash function that remains secure and trustworthy in any sense in the face of (even unsophisticated) adversaries. Most importantly, it is not guaranteed that using two unrelated perceptual hash functions makes finding collisions more difficult. In fact, in many contexts, these adversarial attacks tend to transfer: if they work against one model, they often work against other models as well [3]. To make matters worse, a second, secret hash function can be used only after the collision threshold has been passed (otherwise, it would have to be done on the device, but then it cannot be kept secret). Since the safety voucher is not linked directly to a full resolution photo, the second hashing has to be performed on the tiny "visual derivative", which makes collisions all the more likely. Apple's second hash algorithm is kept secret (so much so that the whitepapers released by Apple do not claim and do not confirm its existence!). This means that we don't know how well it works. We can't even rule out the second hash algorithm being a trivial variation (or completely identical) to the first hash algorithm. Moreover, it's unlikely that the second algorithm was trained on a completely different dataset than the first one (e.g. because there are not many such hash algorithms that work well; moreover, the database of known CSAM content is really quite small compared to the large datasets that good machine learning algorithms require, so testing is necessarily limited). This suggests that transfer attacks are likely to work.

m3kw9 5y ago

So is this going to be used for DDOSing their photo verifying service?