17 ms·
Expanded Protections for Children
- robbiet480 5y agoPrevious discussion at https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741
- ComputerGuru 5y agoWhich would still be on the front page given how many upvotes it had if people hadn’t, for some reason, flagged it.
- Telemakhos 5y agoThe previous discussion was based on a few Tweets. This is Apple's information release, and it is more specific. For me, the first thing on Apple's release is the most important: that they will begin censoring Messages content for all possible nudity, not just hashed CSAM, when the recipient is a minor. That's a step beyond the earlier discussion. Perhaps this same approach can also soon be applied to filtering out the lies and misinformation of the enemies of the Party and to disrupt Emmanuel Goldstein's networks of treasonous domestic terrorists. After all, such misinformation is a plain harm and danger to IngSoc. That Apple is willing to eliminate nudity in private messages gives hope that the Party will soon be able to free all society from doubleplusungood differenthink.
- ComputerGuru 5y agoThe additional information is good reason for this post to also be on the front page. It is not a reason to flag the original and bury that discussion.
- kyralis 5y agoDid you actually read the release? They will pop a warning when the minor sender attempts to send something that is flagged as nudity, or the minor receiver receives something that is, if enabled as part of parental controls for the account. It's opt-in for the account. The CSAM scanning is different, unrelated to the age of account holder, is a condition of iCloud Photos usage only, and is not opt-in (and the only opt-out is to not use iCloud Photos).
- Mindwipe 5y agoThat feels tremendously easy to abuse. As an obvious vector - an abusive husband can set up the account on their partners phone, set them up with a child account and "check" they are not sending anything suggestive. And we have no idea how reliable the scanning is for the sexting piece. A false positive notification could easily get some beaten, even possibly killed. Scanning against a known database of material uploaded to iCloud didn't sound too terrible for me, but this sexting feature is incredibly dangerous.
- Macha 5y agoLet's loop back in an hour and see if this post is still on the front page. I suspect it will have a front page worthy number of votes, but will not be there
- Macha 5y ago33 minutes and it's already off the front page with 161 votes in one hour, compared to items on the front page with 167 votes in 5 hours or 24 votes in one hour. No official source is protecting this one from users flagging it off the front page.
- echelon 5y agoThis is what happens when you abandon individualism and replace it with corporate worship. When you make brands a part of your identity, you can't think about the real world damage being done. If you're on the fence about Apple, and their lockdown of free and open computing wasn't enough, or their strong arming of the mobile commerce economy didn't do it, please let this sink in. This is where we're headed. 1984 is a plausible future. Stop buying Apple and ask your representatives to break up the monopolies. This order came from the FBI and intelligence communities (and probably the CCP and respective organs in China). Not all US lawmakers are beholden to this abuse of power, and they can help us put an end to it. Call them. And stop buying Apple. Fight for your democracy and freedom, lest it be completely wrested from us.
- hoppyhoppy2 5y agoAlso https://news.ycombinator.com/item?id=28075021 https://news.ycombinator.com/item?id=28075021
- celeritascelery 5y ago> Using another technology called threshold secret sharing, the system ensures the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. The threshold is set to provide an extremely high level of accuracy and ensures less than a one in one trillion chance per year of incorrectly flagging a given account. Are they saying that they physically can’t access your iCloud until the threshold is reached, or just that they “promise not to”?
- slaughtr 5y agoIf they can ever access it, then it's always "promise not to".
- samrolken 5y agoIt sounds like this is a feature of the encryption used. Apple will not have enough of the key to decrypt anything unless there are enough vouchers which match.
- ggreer 5y agoApple always has the decryption key for your iCloud data. When law enforcement requests an account's data, Apple decrypts the data and sends it over. Apple's law enforcement guidelines[1] explain: > iCloud content may include email, stored photos, documents, contacts, calendars, bookmarks, Safari Browsing History, Maps Search History, Messages and iOS device backups. iOS device backups may include photos and videos in the Camera Roll, device settings, app data, iMessage, Business Chat, SMS, and MMS messages and voicemail. All iCloud content data stored by Apple is encrypted at the location of the server. When third-party vendors are used to store data, Apple never gives them the encryption keys. Apple retains the encryption keys in its U.S. data centers. iCloud content, as it exists in the customer’s account, may be provided in response to a search warrant issued upon a showing of probable cause, or customer consent. This happens to thousands of accounts every year.[2] 1. See page 11: https://www.apple.com/legal/privacy/law-enforcement-guidelines-us.pdf https://www.apple.com/legal/privacy/law-enforcement-guidelin... 2. https://www.apple.com/legal/transparency/us.html https://www.apple.com/legal/transparency/us.html
- 3327 5y agoBig brother is here. No to Csam but this stuff will not work obviously there will be immediate work arounds.
- netr0ute 5y agoSad, more spyware masqueraded behind "think of the children."
- DSingularity 5y agoAt some point it becomes amoral to do nothing.
- jszymborski 5y agoThat's a false equivalence. Surely there are things that are to be done (and in fact are done) that don't involve spying on all civilians.
- netr0ute 5y agoTo do nothing against what?
- DSingularity 5y agoAgainst the use of your product for something harmful.
- hughrr 5y agoAsk that question to weapons manufacturers.
- stefan_ 5y agoWhich is what, exactly? The irony of this system is that it is obviously only capable to scan for images that are already long in circulation.
- quenix 5y ago... and stop their further propagation. It's kind of like a viral pandemic—just because a virus has been spreading for a while, doesn't mean we should drop any attempts to stop new infections from spreading to five more people each. The more platforms we make unable to keep CSAM on disk, the harder it is for it to circulate in a viral fashion. With this change, all iOS machines become immediately unviable as CP transmission vectors.
- deleted 5y ago[deleted]
- bob1029 5y ago> These efforts will evolve and expand over time. I have no words.
- DSingularity 5y agoI feel like the absolute positions are they easy ones to take. But be honest with me, would you not celebrate an outcome where a system like this eliminates the potential use of iOS for CSAM and in turn causing significant decreases in the abuse of children?
- cultofmetatron 5y agoyou seriously think they are going to stop at CSAM? that just their foot in the door. a Generation ago, they would have used this to identify lgbt people for instance.
- heavyset_go 5y agoYou can be sure it will be used in countries where being LGBT is illegal today, as well.
- echelon 5y agoThis isn't about children! Children mean fuck all to these people. This is the surveillance state building more instrumentation and monitoring into the fabric of society. "Thinking of the children" is the time cherished tradition for pushing through abusive tech. This will be used in Xinjiang and Hong Kong. And soon, it will be used against you. The only relief I have is that Millennials and GenZ aren't having kids and many couldn't give two fucks about them "being protected" from imagined goblins at a cost to their convenience. Maybe this argument won't work anymore.
- bob1029 5y agoNo I would not celebrate such an outcome, because I have been around long enough to be aware of and care about higher order consequences of decisions like these. Have we already forgotten about where fear of terrorism has landed us?
- Syonyk 5y agoDon't shit in my hand and call it chocolate ice cream. "We're going to scan your photos, on your encrypted device, to look for badness. Right now, we're going to claim that's only for the really icky people that nobody is going to defend, but, hey, once the tech is in place, who's to say we can't scan for dank memes and stuff?" I think I'm done with Apple. Sad, really. I was hoping that their bowing to China with iCloud wasn't a sign of what's to come, but apparently it was. They had done such nice stuff with privacy too. Demote my phone to a house phone and go without, I suppose.
- DSingularity 5y agoWould you please describe a solution to finding CSAM which you would find acceptable? It seems to me that on-device scanning for known CSAM content is on the lower end of the privacy-violating spectrum.
- jszymborski 5y agoNo, the lower-end is infiltrating CSAM distribution networks the old-fashioned way.
- gpm 5y agoTraditional police work that respects the principle behind the 4th amendment, no searches of private information, algorithmic or otherwise, directly by the government or otherwise, where you do not have probable cause.
- DSingularity 5y agoWhy can’t algorithms do the police work for some crimes?
- gpm 5y agoAlgorithms can absolutely do the police work for crimes, provided they only perform searches that are supported with probable cause.
- akmarinov 5y agoIf only iCloud backups were encrypted… Also this is US only? What about other children?
- diebeforei485 5y agoThe US has a rigid structure in place (NCMEC, etc), which other countries don't. Also, they have to start somewhere, and they understand the landscape of their home market best. The US also has stronger laws so this is also about protecting Apple.
- DaniloDias 5y agoIf you have young children, better not take pics of them, lest apple inc accuses you of being a predator.
- tptacek 5y agoThey got Mihir Bellare to review (and write a proof for) the private set intersection cryptography pieces of this. https://www.apple.com/child-safety/pdf/Technical_Assessment_of_CSAM_Detection_Mihir_Bellare.pdf https://www.apple.com/child-safety/pdf/Technical_Assessment_... Apple's official proof was done in part by Dan Boneh: https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...
- skinkestek 5y agoI think Moxie came up with the end-to-end encryption technology that WhatsApp and Signal uses. It is great. But it didn't prevent Facebook from trying to abuse WhatsApp users in every conceivable other way including some rather innovative ones as seen recently. Also it didn't prevent Signal from releasing a desktop client with a really nasty XSS bug and the phone client hasn't exactly been without faults either. You and Moxie are both much smarter than me but I foresaw and maybe even predicted (possibly so early it is under another handle) some of Facebooks actions while everyone else was just talking about how E2E-encryption was the big difference. This feels more or less the same: the intent is honorable, the core code is clever and almost unbreakable - but you cannot really trust all of the actors you need to trust for the system to work. We cannot read WhatsApp messages in transit. And, as specified this system sounds good. But WhatsApp happily uploads unencrypted copies to Google Cloud and/or iCloud without your consent if another participant enable backups, and I'm fairly sure we'll see this system containing huge opportunities for abuse by power hungry regimes as well as one or two nasty bugs that has the possibility to utterly destroy a few innocent persons lives.
- ncw96 5y agoA summary of the photo scanning system: - Only applies to photos uploaded to iCloud - Matching against a known set of CSAM (Child Sexual Abuse Material) hashes occurs on-device (as opposed to the on-server matching done by many other providers) - Multiple matches (unspecified threshold) are required to trigger a manual review of matched photos and potential account suspension
- Telemakhos 5y agoYou should reread the first section of TFA, titled "Communication safety in Messages." This goes beyond the scope of CSAM: they're scanning all Messages photos sent to or from a minor's account for any possible nudity, not just CSAM hash-matching.
- eightysixfour 5y agoIt sure seems like this is two different techniques/technologies at work. There is a CSAM deterctor using a known database and then, it looks like, there is a separate model in messages for detecting pornographic content without a known database.
- jeffbee 5y agoIt doesn't sound like it can "detect pornographic content" since the difference between pornography and nudity is not going to be reducible to a coefficient matrix.
- stu2b50 5y agoYeah but that just blurs the image and notifies the parents. Pretty low impact for false positives.
- ncw96 5y agoSorry for the confusion — I was referring to just the CSAM hash feature that uploads results to iCloud. There is also scanning for nudity in the Messages app, but those scans happen on-device and the photos stay on-device even if nudity is detected.
- alpb 5y agoAlternative title: Security researchers voice privacy concerns over Apple's new plan to scan users' data for child abuse images, saying governments will likely increase its scope https://www.ft.com/content/14440f81-d405-452f-97e2-a81458f5411f https://www.ft.com/content/14440f81-d405-452f-97e2-a81458f54...
- jszymborski 5y agoThis will probably result in me ditching the iPhone next upgrade.
- netr0ute 5y agoThis makes me never get an iPhone until they remove this malware infection that they themselves installed.
- TravisHusky 5y agoWow, I am surprised Apple is taking this route. It's not like iCloud was a haven for sharing illegal content. I love the security features of iOS but honestly this may have pushed me to move to an Android device running GrapheneOS. Honestly, any time there is a new policy to "protect children" it is almost always incredibly invasive and it always feels like there is some other motive and "protecting children" is used to scare anyone who tries to question it.
- deleted 5y ago[deleted]
- brightball 5y agoAfter reading the post and as a parent of two kids who are in middle school now...I'm pretty happy with what I see. I didn't expect to be based on the comments I read here before reading the article though. I know a local family who has a daughter who's been in therapy for the last 3 years because she fell victim to the type of thing Apple is discussing in this post. They are firmly advocates for better parent education and oversight, sharing their experience so that other people can hopefully never have to deal with the same thing. They told us about an app called Bark[1] that's supposed to really help with a lot of this stuff and seems inline with what Apple is talking about here. I'm pretty happy to see it will be built in. > The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple. All the parental controls in the world don't prevent the fact that getting your kids a phone in this day and age is a pretty terrifying experience if you know what type of things are out there. > When receiving this type of content, the photo will be blurred and the child will be warned, presented with helpful resources, and reassured it is okay if they do not want to view this photo. As an additional precaution, the child can also be told that, to make sure they are safe, their parents will get a message if they do view it. Similar protections are available if a child attempts to send sexually explicit photos. The child will be warned before the photo is sent, and the parents can receive a message if the child chooses to send it. 1 - https://www.bark.us/ https://www.bark.us/
- webdood90 5y ago> getting your kids a phone in this day and age is a pretty terrifying experience there is a really easy solution to this problem
- brightball 5y agoOh I know. I'm the parent who's kids constantly complain "everybody else has one" and keeps holding out.
- netr0ute 5y agoIf they're complaining about that then they haven't grown up enough to realize they don't need one right now.
- breadsticks3726 5y agoAs a young adult I have to ask this - you guys really fucked up this hard on defending the privacy landscape? How long do I have until a mini space hitler in my pocket that censors and snitches on my activities? Your actions or lack thereof will have grave consequences for billions of people across generations to come.
- cultofmetatron 5y agodamn, I was thinking of getting an iphone. Now I'm thinking of ditching the entire apple ecosystem altogether. The idea that apple can scan files on my system without my consent is pretty sickening. I don't care what its purportedly used for. This is a slippery slope to all sorts of privacy violations.
- oh_sigh 5y agoYou're tacitly giving consent by using their device and services.
- tick_tock_tick 5y agoWell they are now reject that consent and never getting an iPhone.
- rabuse 5y agoSince when did we start allowing companies to own the actual products we purchased?
- nomel 5y agoI imagine it's in the user agreement when you create your iCloud account. The isn't a device thing, it's a web service thing.
- oh_sigh 5y agoHow does them adding a scanning feature mean you don't own your device any more? Do you not own your device because apple puts iOS on your iphone before you buy it?
- kyralis 5y agoDon't use iCloud photos and no scanning occurs. This is a condition of the service, not of the device.
- Syonyk 5y ago
- diebeforei485 5y agoThe first item (on-device nudity detection in conversations) is good and deserves some applause. It's also easy to implement, they could just take an off-the-shelf API and optimize the parameters to their needs, and has minimal privacy concerns because nothing is being reported to Apple. I'm honestly surprised this isn't common already. It's the part about scanning people's photo libraries that folks are (rightly) concerned about.
- RIMR 5y agoIt's absolutely a noble cause, but if Apple is able to scan your iCloud content to figure this stuff out, then your iCloud content isn't secure.
- variant 5y agoAnd how often to noble causes stay noble?
- userbinator 5y ago"The road to hell is paved with good intentions." The gradual but steadily accelerating rise of authoritarianism scares me far more than terrorists, drugs, child abuse, and the pandemic. Unless we push back mightily, it will be a question of when, not if, owning a general-purpose computer that's not controlled by the government or a company becomes discouraged, suspicious, and eventually illegal.
- hypertele-Xii 5y agoDo you honestly think this, what I'm doing right now - posting on HN on a general-purpose computer, on an open source browser and server stack - will become limited to only licenced professionals?
- EvanAnderson 5y agoYes. I absolutely do think that. Companies holding interests in copyrighted works would love to see general purpose computers go away, replaced by "trusted" media players that can't make unauthorized copies or be used to make unsanctioned independent creative works. Totalitarian governments would love to take away general purpose computers to prevent end users from removing surveillance and anonymity. Companies who want to control software markets would love it if all software licensing transactions ran thru their "marketplace".
- userbinator 5y agoStallman predicted this and more: https://www.gnu.org/philosophy/right-to-read.en.html https://www.gnu.org/philosophy/right-to-read.en.html Of course, his suggested solution of open-source didn't quite work out --- because even with the source available, you're still enslaved by things like Secure Boot and other cryptographic jails; and on the other hand, as evidenced by the long history of the cracking scene, and Windows modding in general, not having the source is no big obstacle as long as you can still arbitrarily change any bit. Perhaps what is truly important to software freedom is not "right to read (the source)", but "right to write (anything)".
- 5y ago
- ComputerGuru 5y ago> The device creates a cryptographic safety voucher that encodes the match result along with additional encrypted data about the image. "additional encrypted data" makes it sound like it's encrypted to keep your information safe, when it actually means "encrypted so you can't see what's being sent" and that "only Apple (and people/governments Apple shares decryption keys with) can decrypt it." And what does "additional encrypted data" even mean? I'm guessing it includes a thumbnail. It could include a list of people you shared the image with and their home addresses. The point is, you don't (and can't) know.
- TheJoeMan 5y agoMaybe related to totalitarian regimes like China wanting “origin tracking” which means to know who made the insert Xi meme here
- aunty_helen 5y agoDone with Apple now. I will never own a device that is a revolving door for law enforcement to look into my doings. This constant yo-yo of privacy marketing and bad privacy decisions is too time consuming and mentally stressful. I was waiting to update my 2017 MBP to a M1 16" when they come out and when my iphone 11 is coming up 2 years. These products will no longer be apple products and idgaf if I have to deal with linux desktop and jailbreaking an android to get it done.
- arriu 5y agoI'm happy to see people held accountable for being scummy human beings but this does raise the question of where the line will be drawn and who gets to make that decision.
- sascha_sl 5y agoThis is brilliant misdirection. The crypto is great, but where and how your content is scanned was never the problem to begin with, or at least, a small part. The accuracy of not just the detection rate (there are some outlandish claims of once in billion here), but also the accuracy of the NCMEC database are really the main concern, as well as Apple keeping this system limited to this specific scope. Interesting aside: I once attempted to get access to PhotoDNA, essentially the only insurance against malicious actors abusing upload fields on your website to "digitally swat" you (as has happened to a twitch streamer with an open Dropbox folder), and there is no way you'll get access without a department of lawyers. Why is NCMEC is so protective of an API with rate limits and automated reporting features and then would let Apple ship a bloom filter.
- hackinthebochs 5y agoI remember reading at one point that perceptual hashes used in PhotoDNA could be reversed to reconstruct the image, which is why the database is closely guarded. I assume the bloom filter implementation significantly raises the bar on reversing.
- sascha_sl 5y agoThe basic version of PhotoDNA is an Azure API you pass all your images through directly. Even that is seemingly unobtainable.
- karmicthreat 5y agoFantastic, so is Apple going to notify me if someone is looking at the pics on my phone? No? How about when Apple gets a NSL or other non-disclosable warrant and searches my phone with this tool? Still no? Probably going to be a lot of leaked nudes by Apple soon.
- PragmaticPulp 5y agoI really don’t understand why Apple is doing this. The vast majority of their customers aren’t involved in any of these illegal activities, so it only provides potential downside through false positives. I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos without triggering any number of internet monitoring mechanisms (e.g. honeypots, server logs with their IP address) who would then turn around and upload those photos to their iCloud account. Doesn’t make sense. This is a really strange move.
- jareklupinski 5y agoi'm scratching my head wondering why they bothered announcing it ahead of time? why not just run the scans in the background... (honestly surprised they aren't already) it's not like it would be hard to omit this avenue as your lead during prosecution
- Shank 5y ago> potential downside through false positives I just spent the last hour of my life digging through the material. It seems like they've calibrated the system to have an expected false classification rate of one account in one trillion, per year. Based on the threshold secret sharing math, they won't know anything until a user has a significant collection of CSAM and the secret key is recovered. > I’m also struggling to imagine scenarios where a child predator is clever enough to acquire illegal photos I don't want to be the bearer of bad news, but CSAM has been shared openly on the clearnet on places like 4chan for years. The internet is a pretty wide open search space. Many of the people who download CSAM don't do so from some sketchy underground website. This is why Tumblr, for instance, burned their platform to the ground: lots of people sharing CSAM with no way to detect and stop it (without costing Verizon a ton of money to work on the problem).
- DaniloDias 5y agoAgree. There is no part of this that makes me feel anything good. “We are looking over your shoulder and monitoring your photos… for the children.
- q3k 5y ago
- aerovistae 5y agoIt never occurred to me before reading this post that someday child porn will inevitably be completely vanquished. Open to contradictory views, but my thinking is that someday, eventually, AI will be advanced enough to identify an image's content unambiguously and without error, just like a human user. No human could possibly be unable to correctly identify an image of child porn, and someday algorithms will reach that point too. And once they do....it's probably not a big leap from there for browsers and operating systems to start denying the images altogether, just blacking them out and preventing their transmission over networks. The flipside is that it is impossible for even a human to tell the difference between a 17yo and an 18yo, and moreover impossible for a computer (and arguably for a human) to know whether the user of a device is sending images of themself (i.e. a 17yo sexting with their bf/gf) or whether it's exploitation. That's harder. So for like this new post from apple, this is going to be pretty shitty for high schoolers trying to sext their boyfriends and girlfriends.
- woofie11 5y agoGoodness, I hope not! I run GNU/Linux, and use encryption. Any mandatory technology which could scan my hard drive for child porn could scan it for: - Antigovernment subversive materials - Embarrassing things to blackmail me with - Non-PC communications - Right-wing/left-wing political materials - And so on I'd like to continue to have a computer I control, and I'd like my child to grow up with that freedom too. I'm much more worried about risks to my child's freedom and privacy than the (exceptionally rare but frightening) chance of exploitation for child pornography. As a footnote, many algorithms able to correctly identify child porn would also be able to generate it. https://en.wikipedia.org/wiki/Deepfake_pornography#DeepNude https://en.wikipedia.org/wiki/Deepfake_pornography#DeepNude
- dweekly 5y agoTo the latter point, a minor user transmitting sexual images of themselves may be breaking the law depending on the state and the age of the recipient, so an algorithm may not need to "special case" a "legitimate" reason for the transmission of a sexual image of a minor -- or if the algorithm did, it would need to be very complex to account for the state and ages of both sender and recipient. See e.g. Louisiana's sexting laws: https://www.criminaldefenselawyer.com/resources/teen-sexting-louisiana.htm https://www.criminaldefenselawyer.com/resources/teen-sexting...
- Jaxkr 5y agoThis is incredibly disappointing. The sick criminals that run child pornography rings are not storing their material on iCloud. The "This could be sensitive to view" screen is downright Orwellian. This technology could be used to scan for ANYTHING, completely undermining user privacy. It might just be CP today, but tomorrow it could be screenshots of protest material, whistleblower content, or anti-government memes. I cannot express how sad I am Apple has decided to do this. It doesn't protect children, it won't catch any pedophiles, but it certainly WILL be misused in the future and create a chilling effect on what (politically dissident) content people are willing to store on their phones.
- arriu 5y agoLol there is no way they will be scanning for just one type of thing. That would be a waste of computation. They will tag everything and decide if they want to use it later. Scary times; for exactly the reasons you've called out.
- cyral 5y agoThey already tag everything, just search for any object in your photos library (wether you are on iOS or even Android with Google Photos) and it will find it.
- hypertele-Xii 5y agoI hadn't realized the implications until you brought this up. Thank you. That's... disconcerting?
- angulardragon03 5y agoOn iOS this is done on-device. Apple provides the model(s), and while your phone is charging and asleep it classifies your photos. I can’t find an extended explanation right now, but they say as much under the photos heading on their privacy page: https://www.apple.com/privacy/ https://www.apple.com/privacy/
- 5y ago
- ComputerGuru 5y agoRemember, Apple doesn't actually control the CSAM database (NCMEC). They almost certainly don't (and wouldn't want to) even have access to a reverse mapping between hashes and original images. A/The government(s) could (yes, theoretically - this is cryptography we are purportedly talking about, the onus is on them to prove they can't) easily slip in pictures/screenshots of political materials to have your account flagged. Even if it requires manual human review on Apple's end, the government could still (and has in the past) serve them with a warrant + gag order for "false" matches in the past. Yes, it's all "they could" but with current technical solutions already providing some measure of protection against a corrupt or malicious government from cracking down on its citizens, anything that erodes from that freedom deserves to be held to such a high standard.
- rdtwo 5y agoI wonder if the tiananmen square photo matches one of the Chinese hashes. I bet a bunch of lesser known uyghur photo hashes pop against the database. Is it a publicly available hash database? I wonder if it’s possible to run known controversial stuff and see if it flags?
- judge2020 5y agoSuch images of the Tiananmen square would have been wiped off the face of social media if so - every big social media company uses PhotoDNA or similar to catch likely matches before someone in the real world sees the illegal images on their platform.
- ComputerGuru 5y agoIt has been wiped off the face of social media in China.
- zionic 5y agoIn the press release Apple says they are using hashes of hashes. Apple has no visibility into the original image that generated the hash, so the gov can compromise the list at the source and Apple would have plausible deniability.
- throwawaysea 5y agoSafetyism, particularly with children, is the vehicle that is most often used to enable bad precedent-setting policies, because it can seem virtuous and morally acceptable in that single application. Here, Apple is using a "think of the children" argument to open the door on their intrusion into customers' private data and policing/moderation of the same. Google started doing this themselves recently (https://news.ycombinator.com/item?id=23275308 https://news.ycombinator.com/item?id=23275308) and it is just as unacceptable here. I guess that's the end of me using or recommending Apple products.
- lph 5y agoThis is weird. Apple's own announcement only talks about hash matching, but other reporting (e.g., [0]) talks about a system called 'neuralMatch' that's doing AI on user photos. To me, the privacy implications (and chance of false positives) seems quite different. Quite a discrepancy. [0] https://www.zerohedge.com/technology/apple-plans-monitor-all-us-iphones-evidence-child-porn https://www.zerohedge.com/technology/apple-plans-monitor-all...
- ComputerGuru 5y agoNo, Apple's announcement talks about machine learning to power the iMessage nudity detection algorithm. Machine learning in the context of ruining someone's life (and/or landing them on a government watchlist forever) is a huge no in my book.
- lph 5y agoAh, you're right, it does go into that later. The reporting isn't helping, with sentences like 'the company is rolling out a new machine-learning tool that will scan iPhones for images that match certain "perceptual hashes"'... is that describing a system that classifies new photos, or is it talking about comparing hashes to a known-evil set?
- hughrr 5y agoWhile I believe there is genuine good intent behind this I can’t support apparatus that can be repurposed and can be used to cut me off from my entire digital life due to an arbitrary review process. Not only that I do not consent to using the device I purchased for such measures. I need a grand reset to 2005 levels of technology at this point.
- protoman3000 5y agoCan somebody please explain what the big issue here is? I thought this only applies to iMessage and there are many other possibilities available to communicate securely.
- Zelphyr 5y agoThey are also scanning iCloud documents to match photos against known child abuse photos and reporting any that they find to law enforcement as well as locking you out of your iCloud account.
- kyralis 5y agoNo, they're potentially reporting after some high threshold of matches. They explicitly say that individual matches do not reach that bar, although they do not actually say what "high" is other than the statistical claim.
- smnrchrds 5y agoThey have created and implemented the technology to scan your personal phone for specific documents from a government database and alert the authorities if matches are found. If you don't consider this an issue, you trust your government way too much.
- joshstrange 5y agoThat is not true, they are only scanning photos uploaded to iCloud against the database. The only on-device things being done are detecting nudity in iMessage for minors and alerting the minor/parent. Also something about Siri and detecting searches for CSAM content. Edit: Minor correction, the scanning on your photos does happen on-device but only if you are going to upload it to iCloud.
- smnrchrds 5y agoSo they have implemented the technology. Scanning for photos not being uploaded to iCloud and/or non-photo documents will be a matter of flipping a switch. I am willing to bet serious money that the switch will get flipped sooner rather than later. Perhaps with a facade of government forcing them to do so, so Apple won't look like a bad guy. But in any case, it will get flipped.
- RegnisGnaw 5y agoNote that: 1) its only scanned on upload to iCloud, so if you don't upload then its not scanned 2) (per another article, https://techcrunch.com/2021/08/05/apple-icloud-photos-scanning/ https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...): Most cloud services — Dropbox, Google, and Microsoft to name a few — already scan user files for content that might violate their terms of service or be potentially illegal, like CSAM. So you really can't opt out unless you avoid all cloud photos
- jeffbee 5y agoJust because Techcrunch says it's so, does not mean it is true. My understanding of other services is they might not scan contents that are not shared. Techcrunch does not have first-hand knowledge of this topic and the other clouds are not explicitly saying they do or do not.
- randyrand 5y ago> These efforts will evolve and expand over time. Hard to make any absolute claims about how it works when they admit it will be changing.
- wayneftw 5y agoIf those other services do it on their own computers I'm fine with that. I never expected privacy from them. I don't want my CPU, battery time and network bandwidth to be used for this.
- yellow_lead 5y agoIt's kind of funny they announce this on the first day of Defcon. By next year's conference, hackers will have had a field day with this.
- thih9 5y agoI like that these features are being introduced with something similar to a press release and are being discussed in the open, as well as from the point of view of privacy. It all may still end up being a slippery slope; but with this perhaps the chance is slightly lower (than if Apple introduced these changes without as much of a comment).
- matrix 5y agoThis feature is a big plus for parents. It's hard to appreciate how hard it is to protect kids online if you don't have kids yourself. I get that those without kids will find it intrusive, but it sounds like these feature are opt-in. I consider iOS to be the best platform for kids today. That said, Apple: if you're listening: please tighten up parental controls around time limits and re-loading apps!
- karakot 5y agoI have another idea to protect kids, let's say a kid is moving on a highway and it's speed according to a phone gps is 66mph in 60mph zone, wouldn't be it wise to notify the police about his parents braking a law and endangering a child? There are so many kids dying every year on our roads.[/s]
- bdibs 5y agoDoesn’t this ease the worries from yesterday’s thread? They’ve taken steps to minimize false positives (one in a trillion accounts per year), and it’s not happening on private photos like rumored, only photos that are synced to iCloud (which is what they were doing server side already (?)). I don’t buy the slippery slope argument personally, if you want to get up in arms when they scan for politically motivated content, do so when they actually do that.
- randyrand 5y ago> These efforts will evolve and expand over time. Increases the worry for me. I want to be invest in a company that cares for property rights.
- bdibs 5y agoI took that to mean further protections for children, not expanding the overall scanning. Also isn't it you not caring about their property rights? Should Apple be forced to host CSAM on their servers? Should all businesses be required to?
- Zelphyr 5y agoTo everyone angry at this change, please take a moment to directly let Apple know how you feel. https://www.apple.com/feedback/ https://www.apple.com/feedback/
- jpxw 5y agoFirst off, I don’t think this is some evil plan to kill our privacy. I think this project is done with good intentions, if nothing else. However I think this is an interesting question: how does Apple know that the hashes they’re supplied match CSAM, and not, say, anti-government material? How would they know if the people they got hashes from started supplying anti-government hashes? Apple will only be receiving the hashes here - by design, even they won’t have access to the underlying content to verify what the hashes are for.
- obblekk 5y agoI'll take an unpopular stance. This is the least bad way a backdoor scan could be implemented. At the very least, this makes retroactive scans of a newly banned content difficult without explicit deployment by Apple, and explicit software update from a user. In Western countries, this friction is enough to ensure people will be able to file suit in courts to block the government. In authoritarian governments, it's not, but then, nothing is.
- wyager 5y ago> If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated. Oh, joy, if a robotic bureaucrat nukes my personal data, I have the option to file an appeal that will probably never be looked at.
- tobyjsullivan 5y agoApple has been pro-encryption and pro-privacy lately and is using that as their key differentiator, especially against Google. Various governments, on the other hand, have been very anti-encryption over the past couple decades. Australia's anti-encryption law[0] is just one example but I'm sure there are many others. This presents an ongoing threat to Apple's current strategy. The most common arguments used by governments to justify anti-encryption legislation are child protection and anti-terrorism. I wonder if this is a tactic by Apple to undermine that common argument and pro-actively protect their rights to employ end-to-end encryption and other privacy features. [0] https://fee.org/articles/australia-s-unprecedented-encryption-law-is-a-threat-to-global-privacy/ https://fee.org/articles/australia-s-unprecedented-encryptio...
- diebeforei485 5y agoApple's software is quite buggy. There have been multiple instances of Unicode bugs for example, where receiving a notification or text message containing certain Unicode characters would cause a kernel panic, boot loop, or other fun stuff. People often posted messages containing these Unicode characters in Discord groups (tagging @everyone) for the lulz, because every iPhone user in that group would get a notification containing those unicode characters and then kernel panic. It's only a matter of time until someone finds a bunch of false positives, spams them around for the lulz, and boom people's iCloud accounts are disabled. Even if you believe the theory is mathematically sound, the implementation need not be.
- breck 5y ago> "developed in collaboration with child safety experts" Prove that. Release all git commits and emails and communication and who was involved in these features. I don't see any reason they would not do that, if this was about "child safety". My guess is a portion of these "child safety experts" will have emails ending in "nsa.gov".
- teekert 5y agoIf I understand correctly pictures of my naked children never match any known hashes so I should be ok (1 in a trillion ok that is? What is the expectation value for this?)? Not that I make such pictures but perhaps by accident. It’s not that I find it weird but god I’m afraid they end up beyond my control. I don’t use iCloud (only for MS MFA backups), is this affecting me? I self host everything, what happens when my account is suspended, does my iPhone work? I sure hope we get to learn how effective this has been in say, a year? It’s quite something that my device is going to scan and check my pictures. I’d really like there to be a large, confirmed benefit for the children.
- diebeforei485 5y ago1 in a trillion is not particularly impressive. Aren't there a billion iPhones? And probably 10,000 pictures on each phone on average (people take lots of selfies).
- thw0rted 5y agoWell, since Talking About Statistics Is Hard, you have to look at the exact phrasing from the website: it "ensures less than a one in one trillion chance per year of incorrectly flagging a given account". So, each of those billion accounts has a 1:1T chance of false positive. If I remember my stat 101 correctly, that should translate to a 1:1000 chance of having at least 1 false positive on the planet during any given year. (And remember, even a false positive just means that a human reviews your photos, not that you get reported to the police.)
- threatofrain 5y ago> The Messages app will add new tools to warn children and their parents when receiving or sending sexually explicit photos. > Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit. The feature is designed so that Apple does not get access to the messages.
- malwarebytess 5y agoWoah. I definitely don't want this. I don't want apple trolling through my data. Terrifying. This will be used in other ways. Whenever you hear "protect the children" as an excuse to increase surveillance you know they're up to something horrible. The future is bleak AF.
- nomel 5y agoGoogle has been doing this with their cloud services since at least 2014 (from a quick google search) and Facebook at least 2011. From talking with a founder of a popular image service, this was a requirement a decade ago, for liability reasons.
- malwarebytess 5y agoI don't think you quite understand what Apple is doing. This is absolutely going to lead to deeper use of this kind of anti-encryption tech. third party doctrine loophole to the 4th amendment -- pretty soon every document on your PC will be crosschecked for a litany of things the government deems illegal.
- nomel 5y agoMaybe I'm misunderstanding, but it's scanning cloud hosted images, correct? > allow Apple to detect known CSAM images stored in iCloud Photos. ... Apple only learns about users’ photos if they have a collection of known CSAM in their iCloud Photos account. The only non-cloud hosted scan is for child accounts, with Apple getting no access, just transferring the warning to the parent.
- ogurechny 5y agoIt's always been “think of the children” for three decades in IT, whether it was encryption, anonymity, or access to information. https://en.wikipedia.org/wiki/Four_Horsemen_of_the_Infocalypse https://en.wikipedia.org/wiki/Four_Horsemen_of_the_Infocalyp... Almost every government blocking system started for the sake of the children, then expanded for this or that reason. In some cases, it expanded before starting: notoriously, first sites blocked when Russian government censorship was introduced (to protect our kids from the dangers of the internet!) were well known media entities that had criticized Putin for a long time. The zest was that they were hurriedly blocked even before the relevant acts came into effect. Like, why keep the act after the deed is done? So now there is a government agency that has been blocking oh-so-edgy pictures of nooses on DeviantArt and similar content for years to pretend that they care about the children, and not execute censorship (and sell their extortion services to people who want to remove some information from the Web on the side). Whether that helped any kid is an open question.
- DavideNL 5y agoI think Apple has good intentions. However history shows the next step will be detecting less serious types of abuse, crime, etc. This system will expand slowly, step by step, until it is normalized again. A sad say for privacy and freedom. Obviously any child porn owners will stop using Apple devices, while the rest of us suffer the consequences. I always wonder what would happen if Tim Cook will retire one day and be replaced by someone like Donald Trump. Once all the tools are in place...
- olliej 5y agoCan't wait for this to be extended for all other "criminal" content. In many countries that includes LGBT, pro-democracy, etc materials. In the US weed is illegal at a federal level, so we can get that as well. Woo! And does anyone think this will actually catch any fucking pedos?
- ryukafalz 5y agoSo my concern about this has less to do with the technology itself, and more to do with... What happens when lawmakers make this mandatory? In such a way where the user can't be allowed to remove it? If that ever happens, you've just outlawed phones/computers that allow their users to have ultimate control over the software running on them. No more alternative desktop/mobile OSes. Oops.
- zug_zug 5y agoThis raises some questions. Suppose my hypothetical 17-year-old daughter does send nudes to her boyfriend, and suppose I as a parent am totally okay with this. If this technology isn't using exact-file-matches but heuristics on the file, is there some world where this consensual naked photo gets sent as "evidence" to Apple/NCMEC? "Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC" -- So doesn't "reviewing reports" in this hypothetical mean looking at a child's nudes without consents in this case? The documentation left out the part where this file, stored entirely locally, gets uploaded to apple. Or does this only happen for iCloud?
- shuckles 5y agoOnly if your daughter’s nudes end up in NCMEC’s database, which happens if (among other things) your daughter confirms to NCMEC that they were generated through an abusive scenario and she would like them classified as such.
- zug_zug 5y agoBut that's using "perceptual matching" too isn't it? They say it's configured for a low risk of false-matches, though they don't detail their testing procedure. Presumably they would know almost immediately of false matches, maybe if the odds are really 1/trillion they should commit to disabling the system if any false matches happen at all. Also, is this all configured on the server-side? Suppose Apple has a pervy employee who vouches my daughter's account "looks suspicious," would he/she then be able to override or "manually review" all her pictures? Would I be notified of any such review? There are a lot of unaddressed questions here.
- shuckles 5y agoIt seems like none of your concerns are unaddressed. The following is all from the linked article: 1. Nobody new can see images sent over iMessage under the proposed changes. 2. A perverted employee cannot flag an account as suspicious; only a device can flag itself when a threshold number of photos matching the NCMEC database are found. 3. Accounts which meet the threshold and have the photo thumbnail verified as objectionable will be suspended with an opportunity to appeal.
- arthurcolle 5y agoSo can I just like disable updates right now and not get this malware when it comes out in the future, or will it be some evil OTA update?
- joshstrange 5y ago> These features are coming later this year in updates to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey. As long as you stay on iOS 14/iPadOS 14/watchOS 7/Big Sur then you will never get the on-device features. The photo scanning is happening on photos you upload to iCloud. Other than for malware I'm not aware of Apple ever pushing an OTA (in the sense that the user doesn't get to choose to install the update since almost all updates are technically "OTA"). Even the most recent Pegasus hack fix is in an update you can choose to install.
- option_greek 5y agoI'm going to bet 100:1 that Android and Windows with Mac in tow are going to follow soon with their own implementations (though most likely technology supplied by three letter agencies) in next three years. All that's needed is a law prohibiting unsecured computing devices by government to plugin any 'gaps' (cue: non tpm, locked down bootloader devices)
- summerlight 5y agoI'm not strictly against this if it can effectively help protecting children (which needs completely separate discussions of course). But Apple's usual privacy/security PR masquerade doesn't seems very compatible with this, especially with the sentence "These efforts will evolve and expand over time".
- MonadIsPronad 5y agoPlease, can we just be allowed to control what happens on the hardware we buy? Is that so great a demand?
- aluminum96 5y agoI'm going to contradict the rest of the commenters here: I strongly support this change. The ease of producing and profiting from child pornography has risen tremendously in recent years due to the advent of technology, and it's perfectly ethical for tech companies to account for their impact and attempt to mitigate it. There is no slippery slope from child porn prohibition to political censorship, as so many other commenters fear.
- Mindwipe 5y ago> There is no slippery slope from child porn prohibition to political censorship, as so many other commenters fear. The UK ISPs implemented Cleanfeed, a joint database to block access to child abuse materials, many years ago. Your feelings were pretty common. Since then Cleanfeed has expanded to censor copyright infringing sites by court diktact, and proposed legislation (which will, in some form pass) will effectively use it to block all sites not suitable for under 13 unless they implement age verification and where the guidelines can be updated at will explicitly for policial purposes by the Secretary of State. So it is pretty hard to take your statement seriously where such slope exists and many of us are much further down it.
- ogurechny 5y ago> The ease of producing and profiting from child pornography has risen tremendously in recent years Have I missed something? Running a porn site with underage content and receiving big money from big payment processing systems stopped being an option in the early 2000s after some big busts. Then individuals would try it, and would try to use shady payment sites linked to other crimes. Then child porn, and even traces of past discussions of it, simply disappeared from open web (thanks to Google silently sugarcoating the search results, and hosting and ad businesses being increasingly picky about what they want to see). A typical person won't be able to find any of those producers (apart from obvious scams and honeypots) neither on open web nor on anonymous networks. Heck, even I, having some experience, probably won't be able to find anyone “selling child porn” even if I was told to do so and visit every possible place of discussion. Though I agree that producing all kinds of media with modern technology got easier enormously. A dedicated handheld camera looks like a steam engine today.
- erhk 5y agoSexual predators and sex trafficking are trojan horse reasons for deeper momitoring and privacy violations. No one would argue against child pornograpy being bad and by transitive properties there is an attempt to make arguing against violations of your own privacy equally morally abhorrent.
- resfirestar 5y agoMy main question is whether the system makes it highly impractical to look for 1 to a small number (let's say 5) newly introduced images. We have the secret sharing threshold, how easy is it for the server to change this on the fly? Putting aside the principle for a moment (personally I don't like this because it's a cop in your phone, full stop), if we want to practically prevent government abuse to look for leakers (or whatever category you prefer) then it's helpful if Apple can't reasonably comply with an order to find someone who has just 1-5 specific images.
- jwiley 5y agoWho needs the NSO group? Repressive regime TODO list: 1: create a child safety organization, or require an existing one to accept your images 2: add images of the children of dissidents (or journalists, or leaders of other political parties), photoshopped to be sexually explicit 3: dissident iphones informs on them. Apple turns the information over to the authorities in the host country 4: if Apple pushes back, threaten iphone sales. Or just improve your doctoring. 5: if Apple plays along or doesn't complain, insist on the ability to detect terrorists, criminals, etc. Again, threaten iphone sales, allow Apple to keep the agreement secret. This may only work once or twice, but it's worth a shot! If you make it to step 5, you have a really bespoke, beautifully designed, Apple managed intelligence apparatus. made with love in Cupertino. Maybe this is an overly cynical take, and in addition to the cryptography they have rock-solid, audited governance and internal controls that would prevent it and/or insider abuse. Maybe localities with real data privacy laws (EU) will be able to offer protections to their citizens with fines big enough Apple will begrudgingly agree, so that a repressive regime can't target their citizens as well as citizens in the host country. Maybe this isn't a slippery slope to more exotic forms of surveillance, like scanning your contact list for pedophiles.
- ashleyn 5y agoUpon reviewing TFA it appears this is intended for iCloud users as a privacy-focused alternative to CSAM-scanning the images you upload to iCloud on-server. Rather than leaving images unencrypted on-server so thay they can be checked, the images are checked on your own device prior to upload (to dramatically simplify it). The sell here is that CSAM checking is then done without revealing the image to Apple...unless it's CSAM, and there's enough of it. iCloud was already scanning for CSAM, this time though they do it without seeing what you upload to the server. This is essentially encryption with a backdoor, and the chief objection is basically the same as the Clipper chip/Skipjack. There are privacy issues with the idea of backdoored encryption, as the EFF pointed out. To counter misinformation that is being spread, it's not a little AI cop doing blanket scans on your image library and app caches - it is expressly intended as a part of iCloud CSAM detection. But at the same time it seems trivial for Apple to expand it to that someday, since they just deployed a method capable of doing so trivially. Conceivably they can make the API so that it does a scan when an image is saved to disk. This is not currently what will be done but whether or not they will is anyone's guess.
- Copernicron 5y agoThis is completely unacceptable. Apple is giving themselves the ability to monitor and/or censor anything you do on your phone, no matter if it's end to end encrypted or not. It's not at all a stretch to go from monitoring iMessage for CSAM to monitoring it for gay pornography or any pornography at all. Or monitoring for anti-government sentiments. Sure, they're only scanning for images right now. But there's nothing stopping them from scanning for other things, like Mein Kampf or the writings of Karl Marx. Or some other author who suddenly finds themselves unpopular with the government of the day.
- kleene_op 5y agoIt's almost comical the tendency companies/societies/countries have to invoke absolutely abhorrent concepts in order to justify their future unethical behaviors and shield themselves from any valid criticism. The fact that they raised the children card speaks volumes about the level of fuckery they're gonna deploy on this one.
- wellthisisgreat 5y agoCan someone who understands cryptography well enough please comment, if these hash comparisons can easily be extended to other areas such as contextual analysis of photos or texts? For example would it be easy now to get to the hypothetical scenario where a text containing certain phrases will be flagged if some partner / regulator demands that?
- bob1029 5y agoI posted in this thread a little bit ago. I couldn't find anything coherent to say at the time. Now I have something to say. I will be pushing for our current & future customers to use alternative device vendors as part of our product stack. We have developed a cross platform implementation of our iOS application recently, and this is a great opportunity to start putting it to use. I don't know if Apple is paying attention, but there are enterprise customers out there paying them $300/yr for the permission to use their presumably-secure devices for purposes of running custom LOB applications. Our organization and product are part of the reason you have financial institutions making bulk purchases of iPads and other Apple accessories. If Apple makes it impossible for us to provide assurances to our customers that no data is leaving the devices (aside from agreed-upon application protocols), then we have a serious problem. I am not putting myself in a position where I push a customer to purchase 1000+ iPads for their business if I will then have to answer to their auditors 6 months later on when sensitive customer documents hashes or whatever other horrible things start getting sent to the mothership as part of v2.0 of this shitty idea. Apple even said it themselves - "These efforts will evolve and expand over time.". Is there a special order form if I need to purchase a non-bugged iPad for a classified government setting or very pissed-off enterprise customer? If not, our default recommendation is going to be Microsoft Surface and we will totally deprecate usage of Apple hardware over the course of the next 18-24 months. This is setting a completely unacceptable precedent, and most of our customers I have spoken with today also agree.
- angch 5y agoOne new way for swatting. Send or otherwise inject problematic images to the target, but make sure the target can't see it (image too small, part of larger collage), but their device does. The target gets flagged by Apple for having CSAM. The target gets FBI'd.
- 7373737373 5y agoSimilar things happening in Europe: https://www.patrick-breyer.de/en/chatcontrol-european-parliament-approves-mass-surveillance-of-private-communications/ https://www.patrick-breyer.de/en/chatcontrol-european-parlia...
- aparks517 5y agoDoes Apple have a legal or regulatory requirement to scan iCloud photos for CSAM? I poked around but didn't find a quick answer. Reading through the two linked PDF's, I got the impression that they're aiming to use cryptographic techniques to meet a (possibly self-imposed) requirement to scan for CSAM while revoking their current ability to decrypt photos stored in iCloud. I guess they may want to revoke this ability so that they can no longer be compelled to hand over customer's photos except those for which the requirements of this new system are met. I guess if it results in an increase of the privacy of iCloud content, that's nice as far as it goes. But it does skeeve me out to be reminded that my phone can (and does) paw through my stuff.
- YeBanKo 5y agoUntil this I was typically defaulting to Apple products for privacy reasons. This seems like a major issue to me. This is an ideal spy tool, disguised as an privacy-preserving feature. Any government of a country with a large enough market for Apple, can force Apple to include hashes hashes into localized versions of OS.
- deleted 5y ago[deleted]