13 ms·
We built a system like Apple’s to flag CSAM and concluded the tech was dangerous
- hipsterhelpdesk 5y agoEasy win. Not needed. There’s enough hate for tech already. Apple scrapped it. I wish they would move on.
- 1cvmask 5y agoIn a previous comment on this very same subject on Apple's attempt to flag CSAM I wrote: This invasive capability on the device level is a massive intrusion on everyone's privacy and there will be no limits for governments to expand it's reach once implemented. The scope will always broaden. Well in the article they correctly point out how the scope of scanning is already broad by governments around the world and a violation of privacy by content matching political speech and other forms of censorship and government tracking. We already have that now on the big tech platforms like Twitter that censor or shadow ban contetnt that they as the arbiters (egged on by the politicians and big corporate media) of truth (or truthiness as Colbert used to say in the old show The Colbert Report) label as misinformation or disinformation. Do we now need to be prevented from communicating our thoughts and punished for spreading the truth or non-truths, especially given the false positives, and malware injections and remote device takeovers and hijackings by the Orwellian Big Tech oligopolies. Power corrupts absolutely and this is too much power in the hands of Big Corporations and Governments. From the article in case you need the lowdown: Our system could be easily repurposed for surveillance and censorship. The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching database, and the person using that service would be none the wiser. A foreign government could, for example, compel a service to out people sharing disfavored political speech. That’s no hypothetical: WeChat, the popular Chinese messaging app, already uses content matching to identify dissident material. India enacted rules this year that could require pre-screening content critical of government policy. Russia recently fined Google, Facebook and Twitter for not removing pro-democracy protest materials. We spotted other shortcomings. The content-matching process could have false positives, and malicious users could game the system to subject innocent users to scrutiny. We were so disturbed that we took a step we hadn’t seen before in computer science literature: We warned against our own system design, urging further research on how to mitigate the serious downsides. We’d planned to discuss paths forward at an academic conference this month. That dialogue never happened. The week before our presentation, Apple announced it would deploy its nearly identical system on iCloud Photos, which exists on more than 1.5 billion devices. Apple’s motivation, like ours, was to protect children. And its system was technically more efficient and capable than ours. But we were baffled to see that Apple had few answers for the hard questions we’d surfaced. China is Apple’s second-largest market, with probably hundreds of millions of devices. What stops the Chinese government from demanding Apple scan those devices for pro-democracy materials? Absolutely nothing, except Apple’s solemn promise. This is the same Apple that blocked Chinese citizens from apps that allow access to censored material, that acceded to China’s demand to store user data in state-owned data centers and whose chief executive infamously declared, “We follow the law wherever we do business.” Apple’s muted response about possible misuse is especially puzzling because it’s a high-profile flip-flop. After the 2015 terrorist attack in San Bernardino, Calif., the Justice Department tried to compel Apple to facilitate access to a perpetrator’s encrypted iPhone. Apple refused, swearing in court filings that if it were to build such a capability once, all bets were off about how that capability might be used in future.
- rfd4sgmk8u 5y agoMaybe it is 4d chess. I am very pleased by the pushback on this, in fact given the tech community outcry, this will not happen for another 5 years. Apple bought themselves some time before the beast forced a move. (regardless, i have already made steps to move away from the apple ecosystem. take that tim, see what happens!!!!!)
- knaik94 5y agoOne additional issue that I haven't really seen discussed is how to handle a situation when a false accusation is made. If a person knows the right people who work at these companies, things get sorted out, but I imagine sometimes a person is forced to just handle the consequences. Stepping away from CSAM and going back to something like developer account and apps getting banned on platforms for violating vague "guidelines". It's someone's livelihood that's sometimes destroyed. Demonetization, apps getting banned, payment processors freezing accounts are mostly black box events and most situations aren't even related to crimes dealing with CSAM. If it was something the government made a mistake with, there's legal ways to fight for your rights. There's generally a level of transparency that is afforded to you. It is concerning that people flagged for handling CSAM will not know if they have been manually reviewed. The need to keep the forwarding to authorities a secret is understandable, but a human review before forwarding is only necessary if you expect false positives to begin with. Keeping that flag secret seems like another black box you can't fight as a user. I don't deny the value of catching these criminals, but it throws the idea of due process out the window when the only assurance so far has been "trust us to do the right thing". It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. I am glad this hasn't flown under the radar just because it is Apple who is making these promises. I have heard non-tech people talk about this but there's a lot of misunderstanding.
- hirvi74 5y ago> It's also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers. So, I am not much of conspiracy theorist, but I do like to sometimes fantasize about alternative realities in which they were true. I am not saying the US government had any involvement in Apple's decision, but what if they did? I do agree with your point about how this topic more or less came out of Left-field. It's clear that Apple did not just recently acquire the technological ability to produce this feature in 2021. This feature could have been implemented years ago (like many other companies with a consumer-available cloud storage model already did to some degree). I am just curious if Apple did not really have a "choice" in this matter. Perhaps my monkey brain just want this to be the case.
- rfd4sgmk8u 5y agoI feel somewhat optimistic of the future when many groups saw through this push for on-device scanning for what it was. Damn straight the tech is dangerous.
- TechBro8615 5y agoEvery instance of “government” in this article comes with some qualifier, like “foreign” or “other” – watch out for those foreign governments who might spy on their foreign citizens. Is the implication that this technology could only do evil in other countries? If Apple deploys this in the US, they’re saving the children, but if they deploy it in China, they’re facilitating an oppressive autocracy? Is the US somehow immune from this same threat?
- bsder 5y ago> Is the US somehow immune from this same threat? No, but it's easier to paint China as evil in the US and the US as evil in China if you want people to get the point.
- knaik94 5y agoI think the understanding is that US so far hasn't pushed into law any policy instructing companies like Apple to publicly censor people. Secret surveillance and privacy has been debated, but not freedom of speech. The US has not used the kind of public censorship used by other countries to facilitate an oppressive autocracy. The US government tends to use one of the four horsemen, CSAM, drugs, terrorism, or organized crime as motivation to deploy censorship and undermine privacy but freedom of speech is generally protected. Foreign governments censor things like undesirable political opposition, LGBTQ+ activism, women's rights activism, and historical events like the massacre of protestors. I think the implication is that the technology is likely to do a lot more harm in other countries compared to the harm done in the US, so "it's okay" if it's only deployed in the US in the name of saving children. A lot of people from the US are strongly against the Apple policy regardless.
- noasaservice 5y agoWhen the very mainstream news media is under the same financial umbrella of all the defense contractors, is it no surprise we see the "undesirable political opposition, LGBTQ+ activism, women's rights activism, and historical events like the massacre of protestors" covered up or not even reported on to begin with?
- haspoken 5y agohttps://archive.is/y58Py https://archive.is/y58Py
- dang 5y agoI've re-upped this thread in lieu of https://news.ycombinator.com/item?id=28264032 https://news.ycombinator.com/item?id=28264032, which references this article but is baitier and led to more of a garden-variety thread. The current submission got a surprising amount of upvotes for a post that remained underwater (below the front page): http://hnrankings.info/28238163/ http://hnrankings.info/28238163/. It's on my list to detect threads like that and rescue them. This case will be a hell of an example for testing.
- fortran77 5y ago> Apple’s motivation, like ours, was to protect children. Does anybody really believe Apple's motivation is to "protect children?"
- skinkestek 5y agoI personally believe Apples motive is to protect their customers and by extension themselves.
- ummonk 5y agoCopying over my comment from the last article about this: Nothing about those concerns seems specific to the end-to-end encryption compatible CSAM system they or Apple built... Honestly if I were Apple I'd consider just scrapping the whole thing and doing server side CSAM testing on iCloud photos without rolling out E2E encryption for iCloud photos. It's just not worth the PR blowback.
- baxtr 5y agoI think in reality 99.9% of all people don’t care at all about Apple doing this.
- maverwa 5y agoI‘d guess you could add a few more 9s to that. Almost all people either don’t care or like this. And if you do not fear (or do not understand) the implications and risks I see why they like what apple does. It’s one of the few topics where all of mankind (with very little exceptions) agrees: CSAM is bad! That’s why „we do it for the kids“ always works.
- YLYvYkHeB2NRNT 5y agoWithin my circle, people do not care. They will continue to use Apple products because, "I have nothing to hide." That's what they told me when I brought it up.
- dijit 5y agoMy group is a bit more nuanced: most will likely stick with Apple since the effect is somewhat invisible to them, but this topic brought the question up of “should I stay on iPhone” - which is not a question you want to come up very often if you’re trying to sell these devices.
- ipaddr 5y agoDo they allow you to look through their phones if they give that opinion?
- 1vuio0pswjnm7 5y ago"But Apple has a record of obstructing security research." Any examples besides Corellium.
- IncRnd 5y agoOne of the words that you quoted from the article was linked directly to an example.
- kgeist 5y agoMany of my oppressive country's laws are introduced under pretext "save the children". For example, public discourse of homosexuality is essentially banned, because otherwise underdeveloped minors might get involuntarily exposed to it (and supposedly get psychologically traumatized). Then another law allows banning websites that talk about drugs, LGBT, opposition protests etc. without court order, to save children from being involved in those traumatizing things of course (now it's used to ban opposition sites). And it's hard to argue against it, because you are pushed back, "what, you hate kids? you don't want them to be safe?" It's a clever ugly trick, because most adults are parents, their parental instincts kick in, hearing about all that abuse, and they will support any cause that'll make their kids safer I'm not saying Apple is definitively involved in some shady stuff, but from my perspective, it does look like NSA forced them to do some sort of file scanning backdoor and they came up with this "it's about saving the children" explanation, already successfully in use in oppressive countries.
- read_if_gay_ 5y agoChild abuse, terrorism, money laundering and tax evasion. These are any government’s four horsemen of the apocalypse. According to them, they are roughly the same degree of evil and all deserve the strictest prosecution. But only two aren’t bogeymen.
- jdavis703 5y agoIf this is true, why is it so hard for the US to increase enforcement funding for the IRS while purported anti-sex abuse laws like SESTA/FOSTA are passed with broad bipartisan support?
- Geee 5y agoJust think what kind of power this would give USA when they invade countries like Afganistan. They could easily cancel all people who don't like their presence, and be able to shape the narrative with their propaganda. I'm thinking that maybe this is the actual reason they want tools like this. Afganistan failed because of freedom of speech -> need more tools to limit freedom of speech.
- warkdarrior 5y agoIf only US could have effectively cancelled the Taliban using fake CSAM on their phones...
- zamalek 5y agoThe problem is that Apple have let the genie out of the bottle. With all the, very public, blowback and drama they have created, otherwise ignorant politicians are now aware of what is possible and could start demanding it. Great job, Apple.
- shadilay 5y agoWhat is the CSAM version of SWATting going to be called?
- livinginfear 5y agoI've already written this in a previous comment, however I think it bears repeating: I think Apple have introduced this client-side content scanning technology under the auspices of protecting against CSAM, while its true intention is to allow for the Chinese government to scan citizens' phones for subversive content. I'm convinced that Apple's upper management figure the minimal blowback they're experiencing for this privacy invading technology in the west is worth the expansion of their technology into a much more totalitarian Chinese market. I think that this development has been precipitated by a very visible decline in America's economic, and social position as a world leader. Why not risk this move? America's trajectory is that of almost definite decline.
- legutierr 5y ago> America's trajectory is that of almost definite decline. Well, sure, with that kind of attitude!
- MR4D 5y agoIt doesn’t matter. The horse has left the barn already. Now that every country knows that Apple can do this, they have a pretext for forcing them to do it in the manner of said country’s choosing. That to me is the real loss here.
- nomoreplease 5y agoI believe Jonathan Mayer (one of the authors) is a user/commenter here on HackerNews
- ufmace 5y agoI haven't seen this asked on any of the threads about this yet, but what happens if we identify a few of the pics in their database of Evil Pictures, and send them (presumably from a non-Apple device) to the iPhone of anybody we don't like. Presumably the actual data on the device is still encrypted and can't be accessed remotely, which means we need to trigger a law enforcement investigation which involves seizing the device and compelling the owner to unlock it in order to determine if they actually are a kiddie diddler or something went wrong. Gee, can't see how that could possibly go wrong. /s Meanwhile, the actual kiddie diddlers out there have probably read the 10 million articles published about this by now and know not to use iMessage to trade their pictures, so probably not many of them would actually be caught this way.
- 1MachineElf 5y agoThis method of targeting people you don't like was used heavily against political groups on Facebook during the 2016 US election.
- speleding 5y agoIf you send it via WhatsApp, which has the option to save all pictures to Camera roll, it will get automatically uploaded to iCloud if that's enabled. So no need to trigger a law enforcement investigation yourself in that case.
- PartiallyTyped 5y agoOne important point that I have not seen addressed is the fact that CSAM detection may be able to run in realtime. The orwellian implications of realtime detection are even worse, a constant stream of information about the content a particular user consumes, all generated through the display pipeline because new iPhones have ML accelerators.
- terrorOf 5y agoHow are they sure that what they built is identical to the one Apple built? Too much self-confident?
- drivingmenuts 5y agoA question: if the software that Apple is proposing to use on iCloud is accurate when looking at images sent to/stored on iCloud, how difficult is it to adapt to look at the screen buffer before it’s even saved as an image?
- elisbce 5y agoTerrorists killing innocent people, and Apple did nothing to help. But someone possessing 30 pics of CSAM would be so much more important that Apple needs to implement a whole new surveillance system to track down. The values and priorities in our society are indeed upside down sometimes.
- dustinmoris 5y agoQuick question, isn't the scanning only happening on iCloud? If yes, then honest question: Does anyone actually believe that content uploaded into the cloud doesn't get scanned by big tech companies for whatever latest flavour of "innovation" someone comes up with? For example, GitHub was scanning repos for Co-Pilot. We all know Google already scans everything, including Gmail and Microsoft is the king of "telemetry". What is the difference that scares people about Apple's CSAM scanning? I admit I haven't followed all the news very thoroughly because I usually wait a few weeks to see which outrage sticks and which not. Too much outrage nowadays to keep up with everything so this is an honest question.
- umanwizard 5y ago> isn't the scanning only happening on iCloud? No, the scanning is happening on local devices, but only during the upload-to-iCloud process. That’s the key difference that is upsetting a lot of people.
- dustinmoris 5y agothx!
- amanaplanacanal 5y agoI would say from a legal/moral/political philosophy point of view the difference is that the other implementations do the scanning on their servers, apple wants to do the scanning on my phone. Even if the outcome is the same this feels like a step too far. Even though in reality, I don't want them scanning my data regardless of where it lives. We have already been sliding down this slope, this is just one more step.