6 ms·
The "send known CSAM" attack has existed for a while but never made sense. However, this technology enables a new class of attacks: "send legal porn, collided t
by dannyw 5y ago
The "send known CSAM" attack has existed for a while but never made sense. However, this technology enables a new class of attacks: "send legal porn, collided to match CSAM perceptual hashes".
With the previous status quo:
1. The attacker faces charges of possessing and distributing child pornography
2. The victim may be investigated and charged with child pornography if LEO is somehow alerted (which requires work, and can be traced to the attacker).
Poor risk/reward payoff, specifically the risk outweighs the reward. So it doesn't happen (often).
---
With the new status quo of lossy, on-device CSAM scanning and automated LEO alerting:
1. The attacker never sends CSAM, only material that collides with CSAM hashes. They will be looking at charges of CFAA, extortion, and blackmail.
2. The victim will be automatically investigated by law enforcement, due to Apple's "Safety Voucher" system. The victim will be investigated for possessing child pornography, particularly if the attacker collides legal pornography that may fool a reviewer inspecting a 'visual derivative'.
Great risk/reward payoff. The reward dramatically outweighs the risk, as you can get someone in trouble for CSAM without ever touching CSAM yourself.
If you think ransomware is bad, just imagine CSAM-collision ransomware. Your files will be replaced* with legal pornography that is designed specifically to collide with CSAM hashes and result in automated alerting to law enforcement. Pay X monero within the next 30 minutes, or quite literally, you may go to jail, and be charged with possessing child pornography, until you spend $XXX,XXX on lawyers and expert testimony that demonstrates your innocence.
* Another delivery mechanism for this is simply sending collided photos over WhatsApp, as WhatsApp allows for up to 30 media images in one message, and has settings that will automatically add these images to your iCloud photo library.
- onepunchedman 5y agoSomehow this didn't solidify my trust in Apple! By this standard you can probably mount a half decent defence off "ignorance" if you are even caught sending the colliding material. Add this whole debacle on top of what's going on in the EU parliament and 2021 has been WILD for privacy.
- robertoandred 5y agoLEO is not alerted automatically, where’d you get that idea?
- hypothesis 5y agoThey will be if you collide a low-res image that resembles CSAM. Why would person doing manual review risk his job in case if he’s unsure? Naturally he will just play it safe and report images.
- ec109685 5y agoNot resembles. The adversarial image has to match a private perceptual hash function of the same CSAM image that the NeuralHash function matched before a human reviewer ever looks at it.
- onepunchedman 5y agoDo you have any material on this private function?
- ec109685 5y agoNot beyond the documents Apple has shared. Presumably it will be kept that way given it prevents an adversarial attack against it.
- silisili 5y agoThey'd more or less have to be. Well, not necessarily 'police', but NCMEC. I did work in automating abuse detection years back, and the US govt clearly tells you are not to open/confirm suspected, reported, or happened upon cp. There's a lot of other seemingly weird laws and rules around it.
- robertoandred 5y agoThose laws don’t apply if it’s part of the reporting process. Apple’s stated that they do a manual to decide whether to send a report to NCMEC or not, just like other companies do.
- 5y ago
- EGreg 5y agoSo, everyone is going to turn off their iCloud sync and they won’t be a target anymore?
- croutonwagon 5y agoWell according to reports that are generally the source of these collisions, the hashing code has been on the device since around December 2020 (14.3) https://old.reddit.com/r/MachineLearning/comments/p6hsoh/p_appleneuralhash2onnx_reverseengineered_apple/ https://old.reddit.com/r/MachineLearning/comments/p6hsoh/p_a... If Apple hasn't been honest about WHEN it was built into and added to their code base, why would anyone take their word for HOW its being used, or many of the other statements they are putting in their documents as of yet, at least until they are verified
- heavyset_go 5y agoIt doesn't necessarily mean that it will stop them from being a target, because Apple says this[1]: > This program is ambitious, and protecting children is an important responsibility. These efforts will evolve and expand over time. [1] https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
- copperx 5y ago> This program is ambitious, and protecting children is an important responsibility. These efforts will evolve and expand over time. "Think of the children" is the most recognizable trope in TV and film. They couldn't have phrased that to be more Orwellian.
- copperx 5y agoYes, until they add local scanning to macOS / iOS / iPad OS.
- copperx 5y agoIt seems like I'm not going to sleep tonight. Sure, there is hyperbole in OP's comment (CSAM ransomware and automated law enforcement aren't a thing yet), but we're a few steps from that reality. Even worse, how long will it take until other cloud storage services such as Dropbox, Amazon S3, Google Drive et al implement the same features? Or worse, required by law to do so? This sounds like the start of an exodus from the cloud, at least in the non-developer consumer space.
- onepunchedman 5y agoYeh I was talking in hyperbole, but the possible attack vectors this system enables are so powerful I felt it warranted. Under this system you are able to artificially ddos organizations that verify if CP is sent by sending legitimate, low-res porn whose hash has been modified. You can trigger legitimate investigations by sending CSAM through WhatsApp or through social engineering. You can also fuck with Apple by sending obvious spam. * With regard to the legislative branch, they can even mandate changes to this system they aren't allowed to disclose. Once this system is in place, what is stopping governments from forcing other sets of hashes for matching.
- copperx 5y agoAnd this is just one step away from Apple and Microsoft building this scanning into the OS itself (into the kernel/filesystem code, why not?!). This is beyond insane. Stallman was right. Our devices aren't ours anymore. Now, to be fair, there would be a secondary private hash algorithm running on Apple's servers to minimize the impact of hash collisions, but what's important is that once a file matches a hash locally, the file isn't yours anymore -- it will be uploaded unencrypted to Apple's servers and examined. How easy would it be to shift focus from CSAM into piracy to "protect intellectual property"? Or some other matter?
- onepunchedman 5y agoJup. As others have pointed out, if Apple were willing to lie about the extent of this system and its inception date, why should we suddenly trust that they won't extend its functionality. They themselves explicitly state that the program will be extended, so if this is the starting point I don't think I will be around for the ride. It's a shame as I really love some of their privacy-minded features (e.g. precision of access to the phone's sensors and/or media).
- fay59 5y agoBefore they make it to human review, photos in decrypted vouchers have to pass the CSAM match against a second classifier that Apple keeps to itself. Presumably, if it doesn’t match the same asset, it won’t be passed along. This is explained towards the end of the threat model document that Apple posted to its website. https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
- lamontcg 5y agoYou can still hack someone's phone and upload actual CSAM images. That exposes the attacker to additional charges, but they're already facing extortion and all that anyway. I don't understand the "golly gee whizz, they'd have to commit a severe felony first in order to launch that kind of attack" argument. Don't know why this hasn't already been used on other cloud services, but maybe it will be now that its been more widely publicized.
- sigmar 5y agoWhat happens if someone leaks or guesses the weights on that "secret" classifier? The whole system is so ridiculous even before considering the amount of shenanigans the FBI could pull by putting in non-CSAM hashes.
- iflp 5y agoYou don’t need to have the weights. “Transfer attack” is a thing.
- fay59 5y agoFor better or worse, opaque server-side CSAM models are the norm in the cloud photo hosting world. I imagine that the consequences would be roughly the same as if Google's, Facebook's or Microsoft's "secret classifiers" were leaked.
- sigmar 5y agobut in the cloud setting they have the plaintext of what was uploaded. The attack described above is about abusing the lack of information apple has so they will report an innocent user to the authorities.
- cryptonector 5y agoThe attacker faces no charges because the colliding image can be a harmless meme.