14 ms·
Proposed illegal image detectors on devices are ‘easily fooled’
- commandlinefan 5y agoAnd they can easily generate false positives! The worst of both worlds.
- simondotau 5y agoYou are correct, fooling one of the two hashes Apple places between an image and human review is technically easy, with some prerequisites. (The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges). Talking about how trivial it is on a technical level is, in my opinion, an oversimplification. For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM images. These are images of sexually explicit material where pre-pubescent children are actively involved. Following introspection of my own personal moral compass, it's clear to me that being knowingly in proximity to such material would not be "easy" for me.
- masswerk 5y agoHowever, it may be sufficient to just know the hashes of the offending images – and then craft images accordingly.
- simondotau 5y agoIf the target hash is sufficient, then yes, that would invalidate my argument. Although that still requires one to come into possession of such a hash string, which might not be "easy" as a purely practical exercise.
- nullc 5y agoAll my preimage attack images were created using only the hash of the target, no access to the target image was needed, nor is it obvious to me how it would be helpful at all (at least assuming the goal was for the false positive to look like some unrelated image).
- edgyquant 5y agoBut if you are such a person, who these systems are trying to fight against, it is trivial to defeat the detectors. These systems aren’t designed to hurt the average person.
- simondotau 5y agoI agree, building tools which could lightly mangle CSAM enough to change its hash and defeat detection is trivial. Generating false negatives from a hash algorithm is trivial.
- tsimionescu 5y agoA false positive for a CSAM detector is, by definition, something that is not CSAM, but the detector thinks it is.
- simondotau 5y agoYou are correct that the output of a false positive generator would not be CSAM. But one of the two inputs would need to be CSAM or a derivation of it.
- ClumsyPilot 5y agohash of CSAM is not CSAM do you really think there won't be a large list of CSAM hashes on torrents if you go look for it?
- simondotau 5y agoI doubt you'd find a list of NeuralHash CSAM hashes. And it's even less likely that you'll find a list which includes both NeuralHash as well as Apple's secondary perceptual hash which has not been made public. But maybe you could. Let's say hypothetically, such a torrent does exist. Apple could just rev their hash algorithm in the next version of iOS, immediately rendering those lists useless.
- nullc 5y ago> The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims). > For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images. You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted. You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
- simondotau 5y ago> You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model. What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is: 1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default. 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database. 3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that. 4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them. 5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious. 6. Hope that your target does actually import all thirty images into their iCloud photo gallery. 7. Hope that your hash collsions all work correctly in practice. 8. Hope that Apple's human reviewers screw up. 9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS. 10. Hope that the child safety organisation which Apple reports this to also screws up. 11. Hope that the law enforcement body which this child safety organisation reports this to also screws up. 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target. That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of. Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.
- hellojesus 5y agoAnd exactly zero people are surprised that miniscule, random tweaks to images are imperceptible to humans but obviously trash the hash.
- LocalH 5y agoBut machine learning! Neural nets! Artificial intelligence! All words that have meaning that becomes diluted in public perception, and lead to things sort of becoming a "black box", where we don't really know how the models actually do what they do.
- amelius 5y agoMost machine learning models get it right only in 80% to 90% of cases. So if you have millions of users each with thousands of pictures, you can see how often you run into problems. Even with 99% accuracy, the amount of errors is enormous.
- LogonType10 5y agoAnd each of those errors leads to a SWAT team breaching your front door at 4 AM and pointing rifles at your family.
- charcircuit 5y agoThat's not how it works at all.
- syshum 5y agoWhen the internet was first rolled out to the masses everyone took a basic thing and put "on the internet" after it to make the dotCom bubble Seems history it repeating itself with "Artificial intelligence" with all things done by a computer...
- uoaei 5y agoLots of people would be surprised. Because they aren't educated about how these things work. I'd wager that project managers working on these teams also don't have the understanding (let alone intuition) to judge whether these risks are present, and so will continue fighting for it. The wager is based on the presumption that someone who did understand these risks would pivot or otherwise not allow this initiative to continue. PMs fight so hard because from their position their job is at risk if the project takes a nosedive. But they don't know what they're advocating for a lot of the time. I know quite a few folks like this in FAANG and adjacent spaces.
- joe_the_user 5y agoThis seems very obvious. Neural networks are the "best" image detectors we have. It's documented that they can be easily fooled. The insidious thing is that this can used as a pretext to make the filters flag more images since this would seem to make it "harder to hide illegal images", until Apple just personally scans everything.
- noah_buddy 5y agoThere's a Pandora's box here, too. Bad actors can simply intentionally disseminate memes that match arbitrary bad hashes. Trigger the 20 image (or whatever count) rule for millions of people if you're successful enough.
- kf6nux 5y agoThe system can be attacked at the opposite end as well. Governments can claim hashes are CSAM when the hashes are actually images typically held by political enemies.
- josefx 5y agoI think apple claimed that it requires a database agreed to by at least two governments. Count the countries that discriminate against gay people or authoritarian regimes that regularly cooperate with each other and you quickly realize that Apple just advertised its services to the most morally corrupt and evil people alive.
- MauranKilom 5y agoMatching arbitrary bad hashes for NeuralHash is not sufficient. You also need to somehow match unknown-to-you bad hashes for an unknown-to-you second hash algorithm before you even get to the "30 images" and "human review" parts of the pipeline. That Pandora's box is locked pretty tightly.
- IshKebab 5y ago> in its current form, so-called perceptual hashing based client-side scanning (PH-CSS) algorithms will not be a ‘magic bullet’ for detecting illegal content like CSAM on personal devices Whoever said it would be a magic bullet? > The researchers say this highlights just how easily people with illegal material could fool the surveillance. Ha yes, all you need to do is go to a university computing department and ask them to research algorithms to fool the scanner, and then turn it into an easy-to-use app. Then you chance it with some real CP images. It might work! Or not. Interesting but I don't think anyone at Apple will be shocked by this.
- aidenn0 5y agoPlus, my opinion has been for some time that the point of Apple's image-scanning (for Apple) isn't to detect harmful material in iCloud, it's to project an image of not wanting the harmful material stored in iCloud. Apple has a vested interest in preventing iCloud from becoming "That image storage place for pedophiles." Also, I'm sure that the FBI is probably pretty okay with "we only catch the people who ever, in a single instance, forget to filter their images through this program that fools the filters" since that's probably like 99% of all people.
- Nasrudith 5y agoThat whole notion of "image storage for pedophiles" is deeply stupid and fundamentally illiterate. It is like calling a bank's safe deposit box used to store sealed envelopes "a child porn sanctuary". When dealing with demagogues it is time to go on the offense - there is a reason why McCarthy was damned by the line "Have you no decency?!"
- aidenn0 5y agoThe problem right now is that there really isn't a single demagogue to attack. Also, for the more cynical, here's a headline from 2016 Washington Post: “Joe McCarthy was brought down by attacks on his decency. Trump will lose the same way.”
- simondotau 5y ago
- everyone 5y ago"Proposed illegal image detectors" Meaning the image detectors would be illegal in most countries if actually implemented?
- Ensorceled 5y agoIn English that can be parsed either way. If you are sincere in your misunderstanding of this grammar I'm going to assume you are a non-English speaker or might have a slight development disorder relate to communication.
- jhgb 5y agoThis "misunderstanding" may very well be accidentally correct...at least in the EU.
- kristjansson 5y agoNotably, none of the the algorithms tested in the cited study are Apple's NeuralHash, or comparable algorithms. They look at aHash, pHash (plus a variant thereof), dHash, and and PDQ (used at Facebook for similar applications, apparently). The first 4 date to between 2004 and 2010; the last is more recent, but conceptually similar - the citation [0] for PDQ puts it in the same bucket of 'shallower, stricter, cheaper, faster' algorithms as the first four. No one has proposed any of those as 'illegal image detectors'. Apple's NeuralHash may or may not be robust to the same or different perturbations but the cited study provides basically no new information to inform the conversation its press release wants to be a part of. [0]: https://github.com/facebook/ThreatExchange/blob/main/hashing/hashing.pdf https://github.com/facebook/ThreatExchange/blob/main/hashing...
- nullc 5y agoNeuralhash is so easy to trick that not only can I make small changes that change the hash, I can make small (visible but non-annoying) changes to make an image entirely match an essentially arbitrary hash. E.g. here is a Lena I modified to match Barbra: https://user-images.githubusercontent.com/858454/130310372-d6b2c633-c5d3-4b3e-bc12-d3b2f3104ec6.png https://user-images.githubusercontent.com/858454/130310372-d... https://user-images.githubusercontent.com/858454/130310383-9fcc80ba-117e-4383-9177-0df24bc99f9a.png https://user-images.githubusercontent.com/858454/130310383-9... Simply making the hash not match in any single bit is so trivial that it's hardly worth trying. If you give me a 360x360 image of your choice though I'll make one for you if you still have any doubts.
- simondotau 5y agoI don't think Apple has ever claimed that NeuralHash was hardened against intentional false positives. And when properly considered as just one component of their CSAM scanning architecture, NeuralHash being hardened against intentional false positives isn't necessary for it to serve its purpose.
- joe_the_user 5y agoUh, the gp points both intentional false positives and intentional false negatives (IE, anyone intentionally keeping CSAM can easily hide it while anyone can create an image that triggers the hash and gets human to scan your image and etc).
- standardUser 5y agoIf we're going to surrender our privacy in such an absolute way, can we also install programs on cars to stop them from going over 80mph? I mean, if everything is on the table, let's do something that will save lives.
- ClumsyPilot 5y agoNo, instead we've made cars remotely controllable and programmable so that someone can kill you or pedestrians from their computer, then download some CSAM onto the car's infotainment system for good measure. https://www.theverge.com/2015/7/24/9032179/chrysler-announces-voluntary-recall-hack https://www.theverge.com/2015/7/24/9032179/chrysler-announce...
- literallyaduck 5y agoImage detection is theft. It is either stealing compute cycles or network bandwidth. If you work for a company advancing this technology you are evil and should rethink your life choices.
- edmundsauto 5y agoThis is a pretty hot take! I think by this logic, your comment is also theft - it uses my brain cycles and calories from my hard earned budget to process. (Your second paragraph allows for no context or nuance and is just completely unnecessary.)
- arthurcolle 5y agoHe's right you know
- hda2 5y agoApples and oranges. A person's expectations and rights relating to a discussion on a public forum are nothing like those relating to a device you own. GP's "pretty hot take" is correct.
- bigyikes 5y agoWhat if I’m aware of the image scanning wasting my CPU cycles, just like I’m aware of comments wasting my brain cycles? I don’t think my “rights” related to accessing my OS software is much different than my rights related to accessing discussion forum software. Seems pretty apples to apples to me. (I don’t like image scanning, and I don’t like wasted brain cycles, if that matters)
- hda2 5y ago> What if I’m aware of the image scanning wasting my CPU cycles, just like I’m aware of comments wasting my brain cycles? Before buying the device? Sure, if you want. You would be aware that those wasted cycles and wasted network traffic are part of the price. But what about the rest of us who aren't like you, who bought the device without knowing before hand that this theft would take place?
- GhettoComputers 5y agoIs it a slippery slope fallacy to assume they don't have children's best interest in mind and are using this for other reasons? I see China forcing them to run searches on hardware for them to stay in their market, and with the way Apple kowtowed with iCloud encryption, it would not be a suprise if China already did this and they just need to run more models to train and use for their own marketing reasons. They can get metadata from the devices this way so I don't see anything more private about it: they will not only scan your iCloud pictures, they make your phone run their calcuations at no benefit to you, and its not possible to disable what is going to be used for non optional telemetry if you use iCloud.
- smoldesu 5y agoThere's not a doubt in my mind that China already scans domestically distributed content, otherwise they wouldn't have strong-armed Apple into storing Chinese iCloud data in domestic datacenters. If you really want to put on your tinfoil hat, there's an overwhelming lack of evidence to suggest this isn't happening in America, considering Apple's history of PRISM compliance and inevitable pressure from five-eyes.
- DeathArrow 5y ago>. If you really want to put on your tinfoil hat, there's an overwhelming lack of evidence to suggest this isn't happening in America, considering Apple's history of PRISM compliance and inevitable pressure from five-eyes. I suspect that individuals working with sensitive information in other countries do not use iOS or official Android when they are at work. If US spies on allies it would probably spy 100x more the non alied states. https://www.reuters.com/world/europe/us-security-agency-spied-merkel-other-top-european-officials-through-danish-2021-05-30/ https://www.reuters.com/world/europe/us-security-agency-spie...
- deleted 5y ago[deleted]
- TeeMassive 5y agoThis image detector is literally a permanent search warrant. The State wants "false" positives. Is this a bug or a feature? Yes.
- godmode2019 5y agoHi can someone make a summary how it works, how does it deal with resizing, cropping. Would this also be good for copyright detecting?
- DeathArrow 5y agoI wonder on which datasets did Apple train its "illegal image detection" algorithm.
- jdlshore 5y agoIt’s not an image classifier. It’s an image hashing algorithm that does a server-side comparison to a secret corpus of hashes provided by the National Center for Missing and Exploited Children (or a similar body—I forget the details) and equivalent international organizations. Only hashes that appear in multiple countries’ databases are used.
- ggm 5y agoThis won't stop politicians proposing radical sweeping requirements. We've lost the war on science big time.
- xyzzy21 5y agoI can't image Apple engineers don't know this. But it might have been a negotiating give-away to the (tech-ignorant) government akin to gun control laws banning rifle-mounted bayonets and calling it "gun control progress".