6 ms·
My take on this is that the system is by and large useless. It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can su
by foxfluff 5y ago
My take on this is that the system is by and large useless.
It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system).
So what's left when all the criminals this is supposed to catch have figured it out?
False positives. Only false positives.
Is it really worth turning personal devices into snitches that don't even do a good job of protecting children?
Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.
- Retric 5y agoPerceptual hashes are only used to reduce the search space for human review. Apple doesn’t have images in the CSAM database to do a comparison, but if it’s just a picture of a door their going to reject it. Also, because human review is an expense Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions.
- jdavis703 5y ago> Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions. How can we be sure they won’t cut costs by increasing worker load? I could see them giving each reviewer less time to review individual pictures before passing it on to law enforcement.
- fraa-orolo 5y agoWe can't and they probably will, everyone else seems to already be doing so. There's that Swiss federal police report that only about 10% of NCMEC reports are actually relevant (https://fedpol.report/en/fedpol-in-figures/fight-against-paedophile-crime https://fedpol.report/en/fedpol-in-figures/fight-against-pae...)
- Tagbert 5y agoIf they pass false positives to authorities that will open them up to legal action.
- deleted 5y ago[deleted]
- wyager 5y agoEdit: I incorrectly claimed there wasn’t manual review - see below
- xadhominemx 5y agoWhat is the basis for your understanding?
- wyager 5y agoApologies, I was mistaken: “ Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images. Apple then manually reviews each report to confirm there is a match” The design goal was no human review for individual matches.
- zionic 5y agoApple's human review is largely useless. Trolls will be able to easily use tools slightly modify ambiguous adult porn to collide with a "known CP hash". A human reviewer will see a blurry grayscale derivative of adult pornographic content and hit "report" every time.
- at_a_remove 5y agoThis is the threat model I am looking at. It is number one with a bullet. We have already had a court case where an adult actress had to show up in court and prove that she was adult when experts testified that the images were of a non-adult woman. Baby in the sink? No. But a bunch of the aforementioned? Yeah.
- woofie11 5y agoI don't really want my family photos reviewed by strangers. "Reducing the search space" of photos on my phone isn't an outcome I want to live with. At the time someone is looking at photos of my, my wife/husband/girlfriend/boyfriend, and my kids, they'd better have a darned good reason (e.g. a search warrant). I'd also appreciate if Apple let me know if my false positives were reviewed and found to not be CASM.
- sroussey 5y agoDon’t upload an image anywhere, else it can be reviewed.
- woofie11 5y agoThe whole point of Apple's system is that I don't need to upload an image anywhere. Images from my phone can be stolen and reviewed with no due process, based on proprietary Apple technology.
- partdavid 5y agoThe system as described only submits its safety vouchers when photos are uploaded to iCloud. Not saying it will stay that way, but there are three distinct realms of objection to this system, and it's probably useful to separate them: 1. Objections that in the future, something different will happen with the technology, system, or companies; so that even if the system is unobjectionable now, we should object because of what it might be used for in the future; or how it might change. 2. Objections that Apple can't be trusted to do what they say they are doing, so that even if they say they will only refer cases after careful manual review, or that they will submit images for review that were not uploaded to iCloud, we can't believe them, so we should object. 3. Objections that hold for the system as designed and promised; in other words, even if all the actors do what they say they are doing in good faith and this monitoring never expands, it's still bad. People who have the third kind of objection need to deal with the fact that Apple is basically putting in a system with more careful safeguards than are already in place in many Internet services, even for their "private" media storage or exchange. You likely don't know how the services you use are scanning for CSAM but if the service is at all sizeable (chat, mail, cloud storage) it's likely using PhotoDNA or something similar. I think there are valid objections on all three bases. But there's a difference in saying "this is bad because of something that might happen" and "this is bad because of what is actually happening".
- nullc 5y ago> Perceptual hashes are only used to reduce the search space for human review. False. The Apple proposed system leaks the cryptographic keys needed to decode the images conditional on the match (threshold of matches) of the faulty neuralhash perceptual hash. Matching these hashes results in otherwise encrypted highly confidential data being decodable by apple, accessable on their servers to the relevant staff along with anyone who compromises them or coerces them.
- Retric 5y agoApple can decode the data either way. Their the ones doing the encryption on their servers. There are two basic reasons for this first it’s a backup service which makes end to end encryption risky, but second they also let users share access to their baked up photos. iCloud > photos > shared album.
- tzs 5y ago> It won't catch anything but the dumbest of dumb criminals Dumb is a pretty accurate description of a large fraction of criminals. For the most part you only get smart criminals when you are talking about crimes where you have to be smart to even plan and carry out the crime.
- BiteCode_dev 5y agoGiven the average user don't know what a url is and a pedophile can use the darknet, I'd say criminals are not all dumb.
- nicce 5y agoIt is not so easily comparable. It’s all about your intrests and expectations. Darknet might sound bit complex, but as darknet user, you literally just install different browser.
- pope_meat 5y agoSomething like 40 percent of people use the default browser installed by the os. We're just in an echo chamber of people who know what JavaScript is, and that distorts our perception of the world.
- nicce 5y agoCan you give source for that number? Regardless, browser is like any other app. If that amount of people don’t know how to install apps on their computers, then we have a either real dump people (or just lack of motivation) or great UX design failure in general.
- BiteCode_dev 5y agoUsers are on average incompetent, not dumb. It's a dauntins realisation that sinks in once you have to do support for a web site or app catering to the general population instead of a niche. They don't read, don't know the diff between an app and a web site, don't know right click or drag and drop, think google or chrome is the internet and overall their startegy to solve any problem is as follow: - look for something obvious that seems like the answer but is not scary - click - wait for it - repeat 3 times until ok or give up and call someone or get angry or both Working on a streaming video site really opened my eyes on this one. Most tickets we received were insults, some were incomprehensible garbage, a few were actionable request from someone not understanding anything about their computer. This is nothing like your github ticket. Your parent number are being generous IMO.
- volta83 5y agoThe false positive rate reported in the blogpost for imagenet was 1 in a trillion, and the author concludes that this algorithm is better than they expected.
- foxfluff 5y ago"After running the hashes against 100 million non-CSAM images, Apple found three false positives" So closer to 1/10M. The reporting threshold is made artificially higher by requiring more than one positive. But anyway, that's beside the point. A perceptual hash is not uniformly distributed; it's not a random number. Likewise for photos taken in a specific setting; they do not approach the randomness of a set of random images. So someone snapping a photos in a setting that has features similar to a set of photos in the CSAM database may risk a massively higher false positive rate. It's no longer a million sided dice, it could be a thousand sided dice when your outputs happen to be clustered around similar values due to similar setting. But I can't say I care about false positives. To me the system is bad either way.
- tyingq 5y ago"After running the hashes against 100 million non-CSAM images" They don't say what kind/distribution of non-CSAM images. Landscapes? Parent pix of kids in the bathtub? Cat memes? Porn of young adults? Photos from real estate listings? I suspect some pools of image types would have a much higher hit rate. Edit: And, well "hot dog / not hot dog" is impressive on a set of random landscapes too.
- foxfluff 5y agoWell the same article also claims zero false positives for "a collection of adult pornography." I don't know if the size of that collection is mentioned anywhere. Anyway, I suspect that the algo is more likely to pick defining features of the scene and overall composition (furniture, horizon, lighting, position & shape of subject and other objects) more than the subject matter itself.
- 5y ago
- user-the-name 5y ago> It won't catch anything but the dumbest of dumb criminals This includes the vast majority of pedophiles.
- UncleMeat 5y ago> It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system). Given the reported numbers of illegal images detected by similar systems within Facebook and Google, I think it is very clear that this will catch a lot of illegal content.
- zionic 5y agoFacebook and google are not catching 20m people a year, they're mostly flagging and removing tor/proxy-based throwaway accounts.
- ak391 5y agocan try a web demo of it here on huggingface https://huggingface.co/spaces/akhaliq/AppleNeuralHash2ONNX https://huggingface.co/spaces/akhaliq/AppleNeuralHash2ONNX
- devmor 5y agoThe point of these innovations is never the stated purposes. To catch criminals is an excuse. I would bet a great deal that this system is by and large pressured by state actors for the purpose of creating a new political surveillance tool.
- mayoff 5y ago> It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images Apparently that better way is by using Facebook. Facebook made 20.3 million reports to NCMEC in 2020. https://www.missingkids.org/content/dam/missingkids/gethelp/2020-reports-by-esp.pdf https://www.missingkids.org/content/dam/missingkids/gethelp/...
- foxfluff 5y agoYeah, Facebook's blog post makes me wonder what all the stuff they report actually is. When people say CSAM, I think "kids getting raped" but apparently there's stuff that people find humorous or outrageous and spread it like a meme (and not like pornography). "We found that more than 90% of this content was the same as or visually similar to previously reported content. And copies of just six videos were responsible for more than half of the child exploitative content we reported in that time period." "we evaluated 150 accounts that we reported to NCMEC for uploading child exploitative content in July and August of 2020 and January 2021, and we estimate that more than 75% of these people did not exhibit malicious intent (i.e. did not intend to harm a child). Instead, they appeared to share for other reasons, such as outrage or in poor humor (i.e. a child’s genitals being bitten by an animal)." Based on this, I wouldn't conclude that FB is the platform where people pedos go share their stash of child porn. Their numbers also include Instagram, which I believe is quite popular among teenagers? I wonder how likely it is for teens' own selfies and group pics get flagged and reported to NCMEC. (https://about.fb.com/news/2021/02/preventing-child-exploitation-on-our-apps/ https://about.fb.com/news/2021/02/preventing-child-exploitat...)
- nullc 5y ago> Facebook made 20.3 million reports to NCMEC in 2020. Which appears to have resulted in what... 5 prosecutions?
- throwawaymanbot 5y agoits only been about getting the Surveillance architecture in place... nothing else.
- prirun 5y ago> Is it really worth turning personal devices into snitches that don't even do a good job of protecting children? Yes, because the point is not to protect children. It's to get everyone used to the idea that their content is being monitored. Once that is accomplished, other forms of monitoring can and will be added.
- panta 5y agoExactly. It's a Trojan Horse (https://en.wikipedia.org/wiki/Trojan_Horse https://en.wikipedia.org/wiki/Trojan_Horse) to make more pervasive individual control the new normality. The current motivations are just a pretext.
- wpietri 5y agoI knew a probation officer for sex offenders. They told me that most of them were quite dumb. What the repeat offenders were, though, is dedicated. They had all day to try to avoid getting caught, and the PO had a few minutes per week per offender. It's true that in any arms race, a given advance gets adapted to. This will surely catch a bunch of people up front and then a pretty small number thereafter as the remainder learn to avoid iPhones. But that's how arms races work. You could say that about almost any advance in fighting CSAM.
- woofie11 5y agoI think it's only the dumb ones who get caught. Source: I've met a few white collar criminals.
- wpietri 5y agoProbably some of both. One point of the criminal justice system is to shift incentives such that people with their acts together satisfy their desires without criming. There are plenty of smart, greedy people who just go get an MBA and siphon off value in ways that are technically legal. The risk-adjusted ROI is better.
- woofie11 5y ago> False positives. Only false positives. I really doubt this. In the long term, a few people Apple wants to frame will surely slip into the mix. If Apple didn't want Trump to win, a CASM flag a week before the election might do it.
- numbsafari 5y agoSometimes the best way to catch the really smart or sophisticated criminals is to exploit their less smart and less sophisticated accomplices, co-conspirators, peers, acquaintances, or even their victims.