11 ms·

How can you use it for targeted attacks? This is what would need to happen: 1. Attacker generates images that collide with known CSAM material in the database

• xucheng 5y ago
  > 4. Actually, need to run step 1-3 at least 30 times Depending on how the secret sharing is used in Apple PSI, it may be possible that duplicating the same image 30 times would be enough.
• the8472 5y ago
  > 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times A better collision won't be a grey blob, it'll take some photoshopped and downscaled picture of a kid and massage the least significant bits until it is a collision. https://openai.com/blog/adversarial-example-research/ https://openai.com/blog/adversarial-example-research/
  • FabHK 5y ago
    Remains to be shown whether that is possible, though.
    • mannerheim 5y ago
      It's been done for image classification.
    • nabakin 5y ago
      Unfortunately, it is very likely to be possible. Adversarial ML is extremely effective. I won't be surprised if this is achieved within the day, if not sooner tbh.
    • michaelt 5y ago
      Just yesterday, here on HN there was an article [1] about adversarial attacks that could make road signs get misread by ML recognition systems I'd be astonished if it wasn't possible to do the same thing here. [1] https://news.ycombinator.com/item?id=28204077 https://news.ycombinator.com/item?id=28204077
      • FabHK 5y ago
        But the remarkable thing there (and with all other adversarial attacks I've seen) is that the ML classifier is fooled, while for us humans it is obvious that it is still the original image (if maybe slightly perturbed). But in the case of Apple's CSAM detection, the collision would first have to fool the victim into seeing an innocent picture and storing it (presumably, they would not accept and store actual CSAM [^]), then fool the NeuralHash into thinking it was CSAM (ok, maybe possible, though classifiers <> perceptual hash), then fool the human reviewer into also seeing CSAM (unlike the innocent victim). [^] If the premise is that the "innocent victim" would accept CSAM, then you might as well just send CSAM as an unscrupulous attacker.
        • marcus_holmes 5y ago
          Hmm, not quite: step 1 - As others have pointed out, there are plenty of ways of getting an image onto someone's phone without their explicit permission. WhatsApp (and I believe Messenger) do this by default; if someone sends you an image, it goes onto your phone and gets uploaded to iCloud. step 2 - TFA proves that hash collision works, and fooling perceptual algorithms is already a known thing. This whole automatic screening process is known to be vulnerable already. step 3 - Humans are harder to fool, but tech giants are not great at scaling human intervention; their tendency is to only use humans for exceptions because humans are expensive and unreliable. This is going to be a lowest-cost-bidder developing-country thing where the screeners are targeted on screening X images per hour, for a value of X that allows very little diligence. And the consequences of a false positive are probably going to be minimal - the screeners will be monitored for individual positive/negative rates, but that's about it. We've seen how this plays out for YouTube copyright claims, Google account cancellations, App store delistings, etc. People's lives are going to be ruined because of this tech. I understand that children's lives are already being ruined because of abuse, but I don't see that this tech is going to reduce that problem. If anything it will increase it (because new pictures of child abuse won't be on the hash database).
  • sitharus 5y ago
    So the person would have to accept and save an image that when looks enough like CSAM to confuse a reviewer…
    • the8472 5y ago
      Depending on what algorithm apple uses to generate the "sample" that's shown to the reviewer it may be possible to generate a large image that looks innocent unless downscaled with that specific algorithm and to a specific resolution
    • formerly_proven 5y ago
      Yes, the diligent review performed by the lowest-bidding subcontractor is an excellent defense against career-ending criminal accusations. Nothing can go wrong, this is fine.
      • saynay 5y ago
        I would think there is way easier ways to frame someone with CSAM then this. Like dump a thumbdrive of the stuff on them and report them to the police.
        • Sebb767 5y ago
          The police will not investigate every hint and a thumbdrive still has some plausible deniability. Evidence on your phone looks far worse and, thanks to this new process, law enforcement will receive actual evidence instead of just a hint.
    • hdjjhhvvhga 5y ago
      My WhatsApp automatically saves all images to my photo roll. It has to be explicitly turned off. When the default is on, it's enough that the image is received and the victim has CP on their phone. After the initial shock they delete it, but the image has already been sent to Apple, where a reviewer marked it as CP. Since the user already gave them their full address data in order to be able to use the app store, Appla can automatically send a report to the police.
• spicybright 5y ago
  If you're in close physical contact with a person (like at a job) you just wait for them to put their phone down while unlocked, and do all this.
  • FabHK 5y ago
    Then, with all due respect, the attacker could just download actual CSAM. > If your adversary is the Mossad, YOU’RE GONNA DIE AND THERE’S NOTHING THAT YOU CAN DO ABOUT IT. The Mossad is not intimidated by the fact that you employ https:// https://. If the Mossad wants your data, they’re going to use a drone to replace your cellphone with a piece of uranium that’s shaped like a cellphone, and when you die of tumors filled with tumors, they’re going to hold a press conference and say “It wasn’t us” as they wear t-shirts that say “IT WAS DEFINITELY US,” and then they’re going to buy all of your stuff at your estate sale so that they can directly look at the photos of your vacation instead of reading your insipid emails about them. https://www.usenix.org/system/files/1401_08-12_mickens.pdf https://www.usenix.org/system/files/1401_08-12_mickens.pdf
    • incrudible 5y ago
      Also, this XKCD: https://xkcd.com/538/ https://xkcd.com/538/ People are getting nerd-sniped about hash collisions. It's completely irrelevant. The real-world vector is that an attacker sends CSAM through one of the channels that will trigger a scan. Through iMessage, this should be possible in an unsolicited fashion (correct me if I'm wrong). Otherwise, it's possible through a hacked device. Of course there's plausible deniability here, but like with swatting, it's not a situation you want to be in.
      • aardvarkr 5y ago
        Love the relevant xkcd! And to reply to your point, simply sending unsolicited CSAM via iMessage doesn’t trigger anything. That message has to be saved to your phone then uploaded to iCloud. Someone else above said repeat this process 20-30 times so I presume it can’t be a single incident of CSAM. Seems really really hard to trigger this thing by accident or maliciously
• soziawa 5y ago
  > 6. Apple's CSAM detection then flags these, and they're manually reviewed Is the process actually documented anywhere? Afaik they are just saying that they are verifying a match. This could of course just be a person looking at the hash itself.
  • FabHK 5y ago
    They look at the contents of the "safety voucher", which contains the neural hash and a "visual derivative" of the original image (but not the original image itself). https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
    • hughrr 5y ago
      If it’s a visual derivative, whatever that means, then how does the reviewer know it matches the source image? Sounds like there’s a lot of non determinism in there.
• mannerheim 5y ago
  > 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times Just because the PoC used a meaningless blob doesn't mean that collisions have to be those. Plenty of examples of adversarial attacks on image recognition perturb real images to get the network to misidentify them, but to a human eye the image is unchanged.
  • nyuszika7h 5y ago
    The whole point flew over your head. If it's unchanged to the human eye then surely the human reviewer will see that it's a false positive?
    • mannerheim 5y ago
      No, it's important to point that out lest people think collisions can only be generated with contrived examples. I haven't studied neural hashes in particular, but for CNNs it's extremely trivial to come up with adversarial examples for arbitrary images. Anyway, as for human reviewers, depends on what the image being perturbed is. Computer repair employees have called the police on people who've had pictures of their children in the bath. My understanding is that Apple does not have the source images, only NCMEC, so Apple's employees wouldn't necessarily see that such a case is a false positive. One would hope that when it gets sent to NCMEC, their employees would compare to the source image and see that is a false positive, though.
      • int_19h 5y ago
        Which would still be a privacy violation, since an actual human is looking at a photo you haven't consented to share with them.
        • brokenmachine 5y ago
          That will be clearly laid out on page 1174 of Apples ToS that you had to click to be able to use your $1200 phone for anything but a paperweight.
• cm2187 5y ago
  Don’t imessage and whatsapp automatically store all images received in the iphone’s photo library?
  • vermilingua 5y ago
    iMessage no, WA by default yes, but can be disabled.
    • varajelle 5y ago
      So no need of hash collisions then. One can simply directly send the child porn images to that person via WhatsApp and send her to jail.
      • philote 5y ago
        But then you're searching and finding child porn images to send.
        • GeckoEidechse 5y ago
          I'd be surprised if there won't be a darknet service that does exactly that (send CSAM via $POPULAR_MESSENGER) the moment Apple activates scanning.
• jsdalton 5y ago
  For #4, I know for a fact that my wife’s WhatsApp automatically stores pictures you send her to her iCloud. So the grey blob would definitely be there unless she actively deleted it.
• thinkingemote 5y ago
  the issue is step 6 - review and action Every single tech company is getting rid of manual human review towards an AI based approach. Human-ops they call it - they dont want their employees to be doing this harmful work, plus computers are cheaper and better at We hear about failures of inhuman ops all the time on HN. people being banned, falsely accused, cancelled, accounts locked, credit denied. All because the decisions which were once by humans are now made by machine. This will happen eventually here too. It's the very reason why they have the neuralhash model. To remove the human reviewer.
• Nextgrid 5y ago
  > the NeuralHashes of which, unless I'm mistaken, are not available Given the scanning is client-side wouldn't the client need a list of those hashes to check against? If so it's just a matter of time before those are extracted and used in these attacks.
  • brokenmachine 5y ago
    I think there's some crypto mumbo-jumbo to make it so you can't know if an image matched or not.
• y7 5y ago
  Cross-posting from another thread [1]: 1. Obtain known CSAM that is likely in the database and generate its NeuralHash. 2. Use an image-scaling attack [2] together with adversarial collisions to generate a perturbed image such that its NeuralHash is in the database and its image derivative looks like CSAM. A difference compared to server-side CSAM detection could be that they verify the entire image, and not just the image derivative, before notifying the authorities. [1] https://news.ycombinator.com/item?id=28218922 https://news.ycombinator.com/item?id=28218922 [2] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar...
  • deleted 5y ago
    [deleted]
  • deleted 5y ago
    [deleted]
  • FabHK 5y ago
    Right. So, sending actual CSAM would also work as an attack, but would be detected by the victim and could be corrected (delete images). But a conceivable novel avenue of attack would be to find an image that: 1. Does not look like CSAM to the innocent victim in the original 2. Does match known CSAM by NeuralHash 3. Does look like CSAM in the "visual derivative" reviewed by Apple, as you highlight.
    • avianlyric 5y ago
      Reading the imagine scaling attack article, it’s looks like it’s pretty easy to manufacture an image that: 1. Looks like an innocuous image, indeed even an image the victim is expecting to receive. 2. Downscales in such a way to produce a CSAM match. 3. Downscales for the derivative image to create actual CSAM for the review process. Which is a pretty scary attack vector.
      • FabHK 5y ago
        Depends very much on the process Apple uses to make the "visual derivative", though. Also, defence by producing the original innocuous image (and showing that it triggers both parts of Apple's process, NeuralHash and human review of the visual derivative) should be possible, though a lot of damage might've been done by then.
• fsloth 5y ago
  "How can you use it for targeted attacks?" Just insert a known CSAM image on target's device. Done. I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely. Wait for Apple to find the images. When police investigation is opened, make it very public. Start a social media campaign at the same time. It's enough to fabricate evidence only for a while - the public perception of the individual or the group will be perpetually altered, even though it would surface later that the CSAM material was inserted by hostile third party. You have to think about what nation state entities that are now clients of Pegasus and so on could do with this. Not how safe the individual component is.
  • kreitje 5y ago
    FL Rep Randy Fine filed a report with the Florida Department of Law Enforcement that the sheriff was going to plant CSAM on his computer and arrest him for it. They are even in the same political party. https://www.reddit.com/r/321/comments/jt32rs/fdle_report_between_randy_fine_and_wayne_ivey/ https://www.reddit.com/r/321/comments/jt32rs/fdle_report_bet...
  • JimBard2 5y ago
    Indeed this is not new and has probably been happening for many years already. There are services advertising on dark net markets to “ruin someone’s life” which means you pay some Ukrainian guy $300 in Bitcoin and he plants CSAM on a target’s computer.
  • JKCalhoun 5y ago
    > Just insert a known CSAM image on target's device. Or maybe thirty. You have to surpass the threshold. Also, if Twitter, Google, Microsoft are already deploying CSAM scanning in their services .... why are we not hearing about all the "swatting"?
    • short_sells_poo 5y ago
      Who cares that Bad Company XYZ already well known for not caring about customer privacy does it? Wouldn't you want to push back against even more increasing surveillance? Apply was beating the drum of privacy while it was convenient, wouldn't you want to hold their feet to the fire now that they seemed to do a U-turn?
• gambiting 5y ago
  I don't know why you'd even go through this trouble. At least few years ago finding actual CP on TOR was trivial, not sure if the situation has changed or not. If you're going to blackmail someone, just send actual illegal data, not something that might trigger detection scanners. >> What has changed wrt targeted attacks against innocent people? Anecdote: every single iphone user I know has iCloud sync enabled by default. Every single Android user I know doesn't have google photos sync enabled by default.
  • Marsymars 5y ago
    > Anecdote: every single iphone user I know has iCloud sync enabled by default. Yeah, but a lot of them have long ago maxed out their 5 GB iCloud account.
• eptcyka 5y ago
  I'm sure the reviewers will definitely be able to give each reported image enough time and attention they need, much like the people youtube employs to review videos discussing and exposing animal abuse, holocaust denial and other controversial topics. </sarcasm>
  • shadowgovt 5y ago
    Difference in volume. Images that trip CSAM hash are a lot rarer than the content you just described.
    • eptcyka 5y ago
      I personally am not aware of how the perceptual hash values are distributed in it's keyspace. Perceptual hashes can have issues with uniform distribution, since they are somewhat semantic and most content out there is too. As such, I wouldn't make statements about how often collisions would occur.
• st_goliath 5y ago
  > 7. Apple reviewer.... This part IMO makes Apple itself the most likely "target", but for a different kind of attack. Just wait until someone who wasn't supposed to, somewhere, somehow gets their hands on some of the actual hashes (IMO bound to happen eventually). Also remember that with Apple, we now have an oracle that can tell us. And with all the media attention around the issue, this might further incentivize people to try. From that I can picture a chain of events something like this: 1. Somebody writes a script that generates pre-image collisions like in the post, but for actual hashes Apple uses. 2. The script ends up on the Internet. News reporting picks it up and it spreads around a little. This also means trolls get their hands on it. 3. Tons of colliding image are created by people all over the planet and sent around to even more people. Not for targeted attacks, but simply for the lulz. 4. Newer scripts show up eventually, e.g. for perturbing existing images or similar stunts. More news reporting follows, accelerating the effect and possibly also spreading perturbed images around themselves. Perturbed images (cat pictures, animated gifs, etc...) get uploaded to places like 9gag, reaching large audiences. 5. Repeat steps 1-4 until the Internet and the news grow bored with it. During that entire process, potentially each of those images that ends up on an iDevice will have to be manually reviewed...
  • Grustaf 5y ago
    Do you think Apple might perhaps halt the system if the script get wide publication?
    • dylan604 5y ago
      I've only seen Apple admit defeat once, and that was regarding the trashcan MacPro. Otherwise, it's "you're holding it wrong" type of victim blaming as they quietly revise the issue on the next version. Can anyone else think of times where Apple has admitted to something bad on their end and then reversed/walked away from whatever it was?
      • Grustaf 5y ago
        Do you really care what they "admit"? I thought you were worried about innocent people being framed. Obviously if a way to frame people gets widespread, Apple will stop it. They don't want that publicity.
• anonymousab 5y ago
  > 7. Apple reviewer confuses a featureless blob of gray with CSAM material, several times I find it hard to believe that anyone has faith in any purported manual review by a modern tech giant. Assume the worst and you'll still probably not go far enough.
• deleted 5y ago
  [deleted]
• f3d46600-b66e 5y ago
  What makes CSAM database private? It's my understanding that many tech companies (Microsoft? Dropbox? Google? Apple? Other?) (and many people in those companies) have access to the CSAM database, which essentially makes it public.
  • cyanite 5y ago
    Well, the actual hash table on the device is blinded so the device doesn’t know if an image is a match or not. The server doesn’t learn the actual hash either, unless the threshold of 30 images is reached.
• Johnny555 5y ago
  Actually, need to run step 1-3 at least 30 times You can do steps 2-3 all in one step "Hey Bob, here's a zip file of those funny cat pictures I was telling you about. Some of the files got corrupted and are grayed out for some reason".
• johnla 5y ago
  I don't think this can be used to harm an innocent person. It can raise a red flag but it would be quickly unraised and perhaps an investigation into the source of the fakeout images because THAT person had to have had the real images in possession. If anything, this gives weapons to people against the scanner as we can now bomb the system with false positives rendering it impossible to use. I don't know enough about cryptography but I wonder if there is any ramifications of the hash being broken.
• dang 5y ago
  We detached this subthread from https://news.ycombinator.com/item?id=28219296 https://news.ycombinator.com/item?id=28219296 (it had become massive and this one can stand on its own).
• beiller 5y ago
  Maybe they could install malware that makes all camera images taken using a technique like stenography to cause false positive matches for all the photos taken by the device. Maybe they could share one photo album where all the images are hash collisions.
• visarga 5y ago
  How can we know that the CSAM database is not already poisoned with adversarial images that actually target other kinds of content for different purposes? It would look like CSAM to the naked eye, and nobody can tell the images have been doctored. When reports come in the images would not match, so they need to intercept them before they are discarded by Apple, maybe by having a mole in the team. But it's so much easier than other ways to have an iOS platform scanner for any purpose. Just let them find the doctored images and add them to the database and recruit a person in the Apple team.
• Jcowell 5y ago
  one vector you can use to skip step 3 is to send on WhatsApp. I believe images sent via WhatsApp are auto saved by default last I recalled.
• cirrus3 5y ago
  Are you being serious? #7 is literally "Apple reviewer confuses a featureless blob of gray with CSAM material, several times" 30 times. 30 times a human confused a blob with CSAM?