9 ms·

Apple to scan iPhones for child sex abuse images

112 points by monkeydust 5y ago

pcora 5y ago
It does not seem to be scanning phones, but stuff uploaded to iCloud, which is completely different. The article gives the idea that your device will be scanned.
- tyingq 5y ago
  >"Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes," Apple said. It is scanning phones, but only files that are about to, but haven't yet, be uploaded to iCloud. That scan is happening on the phone itself.
  - mrweasel 5y ago
    That actually make it more okay with me. Apple can't have child pornography on their servers, that would be illegal. However, the fact that they are doing the scanning one the device could indicate that they don't have the ability to do the scans in iCloud. Presumably they can't read even read the images once stored in iCloud, so they have to do it on the device. I don't know if that's the reason, but seems like a reasonable guess.
    - shawnz 5y ago
      Apple actually isn't legally liable for what users upload until it's reported to them. And they are capable of doing the scanning server-side, since iCloud doesn't use end-to-end encryption.
      - tyingq 5y ago
        > since iCloud doesn't use end-to-end encryption Interesting. They say they do, seemingly for many things, though not all[1]. Do you have more info? [1] https://support.apple.com/en-us/HT202303 https://support.apple.com/en-us/HT202303
        shawnz 5y ago
        You are right that some specific features on iCloud do have end-to-end encryption (only those listed under "End-to-end encrypted data" on this page). But the majority of users' sensitive data is not included in that set of features. For example the Photos (what's being affected here), Drive, and Backup features don't use it. Note that any encryption keys backed up using iCloud Backup are therefore effectively not end-to-end protected either. Somewhat misleadingly, this page indicates those features use encryption both "in transit" and "at rest", but Apple controls the encryption keys in those cases, so they are actually not end-to-end encrypted. Here is a report indicating that scanning is already happening on the server side: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... Here is a report about how the FBI specifically pressured them against adding end-to-end encryption to iCloud backups: https://www.reuters.com/article/us-apple-fbi-icloud-exclusive/exclusive-apple-dropped-plan-for-encrypting-backups-after-fbi-complained-sources-idUSKBN1ZK1CT https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
- nix23 5y ago
  >>Before an image is stored onto iCloud Photos, the technology will search for matches of already known CSAM. Apple said that if a match is found a human reviewer will then assess and report the user to law enforcement. >>Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes
- MonkeyClub 5y ago
  Exactly, only "private" cloud data will be scanned instead, which is industry-standard practice for any self-respecting cloud provider anyway. It's a wonder how Apple wasn't doing it already. In any case, this will be automated, rather than some poor Tier-1 pouring over iCloud Photos. So only the guilty (and the false positives) would worry.
  - 0xy 5y ago
    This is false, the scanning occurs on the phone. Plus, as has already been discussed at length, the NCMEC database is loaded with false positives. The "nothing to hide" argument tends to fall apart when the database being used against you is full of legal imagery (which often isn't borderline or pornographic at all -- some of the flagged images literally don't show people). Slippery slope to non-CSAM material? That ship has sailed already. The databases are a mess. From day 1, it detects non-CSAM.
    - MonkeyClub 5y ago
      And whatever would make us believe that as users we should expect a difference between a company's servers and the same company's devices? Would the device belong to us, or be covered by some misguided sense of ownership and privacy? The garden was already walled, and the name on the gate wasn't the user's to begin with. Now, where's my rusty yet trusty Nokia? Oh, wait, it doesn't have VoLTE...
  - oauea 5y ago
    > So only the guilty (and the false positives) would worry. If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement. You're not guilty, are you?
    - MonkeyClub 5y ago
      My point exactly. What is privacy in the face of CSAM, after all. /s (if not already clear enough)
swiley 5y ago
This pushed me over the edge to get the last remaining thing I use my iPhone for working on my Pinephone: maps. Marble (from KDE) is really good and even has offline routing support. I think I'll buy a GPS unit just in case (I was planning on getting one for a boat, unfortunately I haven't seen any that do both sea charts and turn by turn but I'm all ears if people know of any.) My iPhone is formatted with an empty battery and in a drawer now. I worry for other people though, the privacy implications of this are insane. This is a huge misstep even for Apple.
oktoberpaard 5y ago
I’ve got the feeling that they’re already doing this on their servers for photos that are uploaded to iCloud (which I believe is pretty standard for services like this). By moving this scan to the devices themselves they can offer end to end encrypted storage without losing the ability to scan for child sex abuse images. Hence, I wouldn’t be surprised if they launch end to end encrypted iCloud storage within the next year.
- new299 5y ago
  Yes they already seem to scan iCloud: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... But I seriously doubt they will make iCloud end-to-end encrypted. If they wanted to do that they would have done so already. And why not announce it at the same time, which might give them an overall PR win? But we shall see I guess!
  - em500 5y ago
    Moreover, if they make it truly end-to-end encrypted, they wouldn't even need to scan on the server side (since they'd only be storing random bytes from their perspective).
- tpush 5y ago
  That's pretty much the theory of a lot of people on this. On-device CSAM scanning as a trade-off for making iCloud backups, photos etc. e2e encrypted.
dubcanada 5y ago
I am not sure I fully understand the concern people are raising. Everyone already doing this, Google Photos/Drive, Dropbox, Youtube, etc. The only difference is Apple just started doing it? Edit: people are downvoting for asking a question...
- beebeepka 5y ago
  It's done locally now. Hardly the same thing. And sounds extremely illegal to me
- XorNot 5y ago
  Apple are proposing to do it on your phone. There is no possible way they are not going to do it to everything on your phone. That's a big policy change, since they're circumventing their own E2EE using their super-admin powers to do it. The barrier when uploading to unencrypted services is that you are at least aware you're granting access. This is invading the phone to do it anyway - that's quite different. But more importantly: when flagged this is going to send the results to human reviewers. Which has not been elaborated on: there's no way human reviewers can screen things without being sent a copy of the image in question. Which means, a neural network system is going to randomly send your personal images to human reviewers - and it will, largely, be false positives. Note that Apple are not discussing anything about the reliability of this system: have they run it against a sample set of normal images? How many did it flag? Because, if they give a number, it'll be pretty easy for people to realize that some % multiplied by the number of phones and average number of images per phone, and I'm willing to bet what you get is: the system will false positive at least (and probably more then 1) personal photo per iPhone user, and send it to human reviewers. And that might get people's attention: certainly more then just, unfortunately, us techies. EDIT: And let's talk about these human reviewers: these are not random citizens seeing something and wondering if they should be concerned. The context these people are going to be given is possible child abuse image. This is not a neutral review process - at all.
- martimarkov 5y ago
  The concern is I bought an iPhone based on the promise of privacy of my device. Now there is a monitoring tool on my phone looking at photos for child porn. I get a false positive and some rando from Apple is now looking at my personal photos. Why? Normalisation of this invasive behaviour is not okay. Apple specifically argued they are not Google and do not invade your privacy.
qubex 5y ago
I’m a nudist. I’m wondering whether we’ll be a persecuted minority (as opposed to merely derided comic relief).
- swiley 5y ago
  Judging by how Apple has handled the App Store you'll certainly be reported.
- monkeynotes 5y ago
  What about people with young children? I know my parents have a good number of 35mm shots of me in the bath with my brother and sister when we were babies.
  - zimpenfish 5y ago
    > What about people with young children? It's not matching an AI's interpretation of "is this a naked child?" though - it's specifically only matching perceptual hashes against a known database of CSAM percpetual hashes provided by NCMEC which will then be manually reviewed if there's more than a threshold of matches.
    - aaomidi 5y ago
      So we should assume any and all of our photos are probably going to be looked at by some random people? What's stopping said people from leaking photos?
      - zimpenfish 5y ago
        > So we should assume any and all of our photos are probably going to be looked at by some random people? No? Not unless they're matching the perceptual hashes of the CSAM provided by NCMEC and in that case, sorry, my sympathy does not extend that far. If anything, this makes it harder for random Apple people to look at your iCloud photos if moving the matching step down to the phone means iCloud photos are going to be end-to-end encrypted, right? (Which is not a given, of course, just a theory that some people have about why it's moving down to the phone level.)
beebeepka 5y ago
Stuff like this is triggering my whataboutism alarm as not a day goes by without the self proclaimed leaders of the "free world" attacking certain governments on nothing but suspicion and "intelligence". Before 2013, the more cynical ones among us were simply branded conspiracy nuts and crackpots. If anyone believed Apple's lies about privacy, that's on them. The solution is to buy more smart stuff from American companies. They are the only ones that can be trusted
whiddershins 5y ago
Does it matter that they are only doing this for images uploaded to iCloud?
ballenf 5y ago
If we don't own the phone, we have no basis to insist on a search warrant before a search. So moves to make it obvious that Apple really owns the phone and we're just pushing buttons on it, undermines any efforts to establish legal precedents around searches and seizures.
- gruez 5y ago
  >If we don't own the phone, we have no basis to insist on a search warrant before a search. Thats not how it works. Your house is still protected from search and seizure even if you rent it.
oakfr 5y ago
FWIW Microsoft already does this for photos you store on OneDrive: https://www.theregister.com/2014/08/07/ms_tip_off_child_abuse_arrest/ https://www.theregister.com/2014/08/07/ms_tip_off_child_abus...
- oauea 5y ago
  That's great but irrelevant. This discussion is about scanning them locally on your device.
  - oakfr 5y ago
    Indeed. Thanks for pointing this out. Scanning on device is a different story.
    - deleted 5y ago
      [deleted]
  - zimpenfish 5y ago
    Well, it's semi-relevant, isn't it? Because it's only scanning photos that are being uploaded to iCloud Photos - they've just moved the scanning portion down one level. If your photos never go to iCloud, they won't be scanned.
    - cageface 5y ago
      So what's the point of doing this client side then? It seems like they're deliberating paving the way for doing this on all client side material.
      - zimpenfish 5y ago
        > So what's the point of doing this client side then? Not a clue. Maybe they're going to make iCloud Photos fully E2E and this is the only way they can keep the CSAM scanning alive? Or maybe they just want to avoid it arriving on their servers at all (but I don't think anything has said matching CSAM photos will not be uploaded?) Or they just want to save some CPU on the iCloud side by punting that down to the phones themselves? (Or, yeah, this is just the precursor to doing more client-side scanning and the slippery slope cassandras will be proven correct.)
markus_zhang 5y ago
Time to use a dumb phone. I realized recently that I can uninstall most apps on my modern phone and still live without any issue. That's the privilege of immigrating to a foreign country as friends and family now don't expect you to reply as soon as possible.
vegetablepotpie 5y ago
How likely are perceptual hashes to give a false positive? If I take a picture of a tree, how likely is it that a few pixels are going to line up just right in a hashing algorithm and say it might be child porn? How likely is it that law enforcement is going to understand the limitations of this technology? How likely is it that the judicial system will understand the limitations of this technology before issuing a search warrant? I can see law enforcement showing up at my door one day demanding to have a look around, and I would have no idea why they’re there, but they’ll want to look through all my personal belongings. Worse yet, I might come home from work one day, see my windows broken, see my place has been ransacked and my computers are missing. I would call the police to report a burglary only to hear that I’m under investigation and they need me to give them the key to decrypt my hard drives. I feel like I need to do some risk analysis on my digital life. I need my phone for 2FA apps for investment accounts. If I use a dumb phone, I’m stuck with SMS, which is easy to hack. But if we use iPhones, there is a small but costly risk of a completely erroneous investigation. How do you balance these without knowing the probability and impact of the later?
- markus_zhang 5y ago
  EDIT: OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. -------- At least it's a lot more possible to mis classify a photo than a sms message. Plus as I mentioned in another post, what if you have a young kid and you happen to shoot a photo when he or she is not fully dressed?
  - hermannj314 5y ago
    Why would that photo match a list of hashes of known CSAM though? This technology at Apple is very much about matching hashes.
    - markus_zhang 5y ago
      Thanks, original post edited.
ByteWelder 5y ago
A better title would be: Apple to scan iPhones for banned images. SwiftOnSecurity: > Just to state: Apple's scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn't know what a child is. (from https://twitter.com/SwiftOnSecurity/status/1423383256003747840 https://twitter.com/SwiftOnSecurity/status/14233832560037478...)
- hef19898 5y ago
  Nice. Well, Apple had me consider a switch from Pixel phones when they prioritized privacy over Facebook. Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google. And they were that close to become an acceptable big tech co regarding user privacy...
  - zimpenfish 5y ago
    > Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google. Don't Google already do this for images you store in their cloud?
    - rvz 5y ago
      yes. [0] https://www.theverge.com/2014/8/5/5970141/how-google-scans-your-gmail-for-child-porn https://www.theverge.com/2014/8/5/5970141/how-google-scans-y... [1] https://torrentfreak.com/google-drive-uses-hash-matching-detect-pirated-content/ https://torrentfreak.com/google-drive-uses-hash-matching-det...
    - temeritatis 5y ago
      Sure, but so does icloud and onedrive. And while i don't like that either, at least one can opt out of it. The only way to realistically opt out of this new system is to not use an iOS device. And i'm just wondering how long until they want to scan your mac as well.
markus_zhang 5y ago
EDIT: OK I didn't read the article, that's my fault. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. ---------- I want to add that this may be a serious issue for people with young kids. Imagine you shoot a video/photo of your kid with little dressed on. Now what happens? Are we supposed to fully dress kids because we have no idea what Apple will do after the scan?
- tunnuz 5y ago
  This.
  - Jowsey 5y ago
    Not this. Read the damn article, people. It clearly says it's matching pre-determined hashes, nothing to do with AI. You're making this worse than it has to be.
    - markus_zhang 5y ago
      OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud.
- tpush 5y ago
  It checks images against an on-device database of known CSAM, your photos should be fine.
  - markus_zhang 5y ago
    OK. Another argument: what if a virus download the pic for you?
    - clifdweller 5y ago
solarkraft 5y ago
Looks like that concludes my otherwise very nice time in the Apple ecosystem. Luckily Linux is coming along well both on phones and the new Macs.
bjarneh 5y ago
Slippery slope...
gerhardi 5y ago
So would this make a good vector for black mailing? Also I guess many parents might have a lot of pictures of their small children playing around naked.. at least where I live.
cheese_van 5y ago
Powerful corporations aligning interests with law enforcement by assuming certain LE roles seems a recipe for corporate dystopian oligarchy. This opens, for Apple, a catalog of answers for "How can we make a buck," and none of the answers will be friendly to privacy or freedom.
2OEH8eoCRo0 5y ago
Let me get this straight- the iPhone will now scan your iCloud account and compute hashes for images. These hashes are then compared against a known database of child abuse images. When a certain threshold of "hits" is reached then steps are taken to notify authorities? There seems to be little chance of false positives here. I keep seeing references to machine learning algos and that's not what they are doing at all. I do still lament losing yet more control though. A device that I buy should not be working for somebody else, for better or worse. CSAM is not new and is already in widespread use. The difference here just seems to be that your device itself will sell you out. Edit: Follow up question- They run this on images before they are uploaded to iCloud. Why has Apple chosen to do this on-device rather than just run it in iCloud? They say for privacy reasons: 'Apple’s method of detecting known CSAM is designed with user privacy in mind.'
- whitepaint 5y ago
  Isn't this very very silly? You just need to change one pixel and the hash will be different.
  - deleted 5y ago
    [deleted]
  - krapp 5y ago
    No, perceptual hashes don't work that way. It's a comparison of similarity to a target, not just making an MD5 hash of an image. Here's Apple's documentation on it : https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
    - whitepaint 5y ago
      Oh nice, didn't know about this.
    - 2OEH8eoCRo0 5y ago
      Very helpful. Thanks! Will give this a read.
syntaxfree 5y ago
So wait, I have pictures of my baby son’s poop to send the pediatrician — which show what will eventually be his naughty bits, as he becomes sexual (in due time, years from now) I actually don’t have many nudeish pics of him because it’s very cold right now, but we take dozens and dozens of daily pictures so — but in the summer… when maybe he’s going to the kiddie pool grandpa has…
merricksb 5y ago
Earlier discussions: Apple's plan to “think different” about encryption opens a backdoor to your life – https://news.ycombinator.com/item?id=28079171 https://news.ycombinator.com/item?id=28079171 – Aug 2021 (748 comments) Expanded Protections for Children - https://news.ycombinator.com/item?id=28078115 https://news.ycombinator.com/item?id=28078115 - Aug 2021 (353 comments) Apple plans to scan US iPhones for child abuse imagery - https://news.ycombinator.com/item?id=28075021 https://news.ycombinator.com/item?id=28075021 - Aug 2021 (368 comments) Apple enabling client-side CSAM scanning on iPhone tomorrow - https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741 - Aug 2021 (716 comments)
deleted 5y ago
[deleted]
White_Wolf 5y ago
what stops apple from checking if you have any content they don't like of like non-pc memes and such and flag you in some databse for futher monitoring/enquiry by pc police?
- clifdweller 5y ago
  that is the big concern that in future you share pepe frog meme or in say russia a pride flag and it gets marked as terrorist image and they report you to _____ or block you from posting on ______
me_me_me 5y ago
I wonder if this is secretly brilliant stunt. Blast on every channel message that your product will actively look for children porn store on it. Stir great debate. And then quietly drop the proposed feature. Your product will be viewed as unsafe to use by the pedophiles maybe even whole apple ecosystem will.
jadecolour 5y ago
I guess it's just a disguised protest against the government by Apple, that they have to give up their stance on privacy and make the world know. Yes, children is so precious and helpless. I think in order to protect them you all have to give up your liberty, so to protect the future of human race.

112 points by monkeydust 5y ago

It does not seem to be scanning phones, but stuff uploaded to iCloud, which is completely different. The article gives the idea that your device will be scanned.

tyingq 5y ago
>"Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes," Apple said. It is scanning phones, but only files that are about to, but haven't yet, be uploaded to iCloud. That scan is happening on the phone itself.
- mrweasel 5y ago
  That actually make it more okay with me. Apple can't have child pornography on their servers, that would be illegal. However, the fact that they are doing the scanning one the device could indicate that they don't have the ability to do the scans in iCloud. Presumably they can't read even read the images once stored in iCloud, so they have to do it on the device. I don't know if that's the reason, but seems like a reasonable guess.
  - shawnz 5y ago
    Apple actually isn't legally liable for what users upload until it's reported to them. And they are capable of doing the scanning server-side, since iCloud doesn't use end-to-end encryption.
    - tyingq 5y ago
      > since iCloud doesn't use end-to-end encryption Interesting. They say they do, seemingly for many things, though not all[1]. Do you have more info? [1] https://support.apple.com/en-us/HT202303 https://support.apple.com/en-us/HT202303
      - shawnz 5y ago
        You are right that some specific features on iCloud do have end-to-end encryption (only those listed under "End-to-end encrypted data" on this page). But the majority of users' sensitive data is not included in that set of features. For example the Photos (what's being affected here), Drive, and Backup features don't use it. Note that any encryption keys backed up using iCloud Backup are therefore effectively not end-to-end protected either. Somewhat misleadingly, this page indicates those features use encryption both "in transit" and "at rest", but Apple controls the encryption keys in those cases, so they are actually not end-to-end encrypted. Here is a report indicating that scanning is already happening on the server side: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... Here is a report about how the FBI specifically pressured them against adding end-to-end encryption to iCloud backups: https://www.reuters.com/article/us-apple-fbi-icloud-exclusive/exclusive-apple-dropped-plan-for-encrypting-backups-after-fbi-complained-sources-idUSKBN1ZK1CT https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
nix23 5y ago
>>Before an image is stored onto iCloud Photos, the technology will search for matches of already known CSAM. Apple said that if a match is found a human reviewer will then assess and report the user to law enforcement. >>Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes
MonkeyClub 5y ago
Exactly, only "private" cloud data will be scanned instead, which is industry-standard practice for any self-respecting cloud provider anyway. It's a wonder how Apple wasn't doing it already. In any case, this will be automated, rather than some poor Tier-1 pouring over iCloud Photos. So only the guilty (and the false positives) would worry.
- 0xy 5y ago
  This is false, the scanning occurs on the phone. Plus, as has already been discussed at length, the NCMEC database is loaded with false positives. The "nothing to hide" argument tends to fall apart when the database being used against you is full of legal imagery (which often isn't borderline or pornographic at all -- some of the flagged images literally don't show people). Slippery slope to non-CSAM material? That ship has sailed already. The databases are a mess. From day 1, it detects non-CSAM.
  - MonkeyClub 5y ago
    And whatever would make us believe that as users we should expect a difference between a company's servers and the same company's devices? Would the device belong to us, or be covered by some misguided sense of ownership and privacy? The garden was already walled, and the name on the gate wasn't the user's to begin with. Now, where's my rusty yet trusty Nokia? Oh, wait, it doesn't have VoLTE...
- oauea 5y ago
  > So only the guilty (and the false positives) would worry. If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement. You're not guilty, are you?
  - MonkeyClub 5y ago
    My point exactly. What is privacy in the face of CSAM, after all. /s (if not already clear enough)

swiley 5y ago

This pushed me over the edge to get the last remaining thing I use my iPhone for working on my Pinephone: maps. Marble (from KDE) is really good and even has offline routing support. I think I'll buy a GPS unit just in case (I was planning on getting one for a boat, unfortunately I haven't seen any that do both sea charts and turn by turn but I'm all ears if people know of any.) My iPhone is formatted with an empty battery and in a drawer now. I worry for other people though, the privacy implications of this are insane. This is a huge misstep even for Apple.

oktoberpaard 5y ago

I’ve got the feeling that they’re already doing this on their servers for photos that are uploaded to iCloud (which I believe is pretty standard for services like this). By moving this scan to the devices themselves they can offer end to end encrypted storage without losing the ability to scan for child sex abuse images. Hence, I wouldn’t be surprised if they launch end to end encrypted iCloud storage within the next year.

new299 5y ago
Yes they already seem to scan iCloud: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... But I seriously doubt they will make iCloud end-to-end encrypted. If they wanted to do that they would have done so already. And why not announce it at the same time, which might give them an overall PR win? But we shall see I guess!
- em500 5y ago
  Moreover, if they make it truly end-to-end encrypted, they wouldn't even need to scan on the server side (since they'd only be storing random bytes from their perspective).
tpush 5y ago
That's pretty much the theory of a lot of people on this. On-device CSAM scanning as a trade-off for making iCloud backups, photos etc. e2e encrypted.

dubcanada 5y ago

I am not sure I fully understand the concern people are raising. Everyone already doing this, Google Photos/Drive, Dropbox, Youtube, etc. The only difference is Apple just started doing it? Edit: people are downvoting for asking a question...

beebeepka 5y ago
It's done locally now. Hardly the same thing. And sounds extremely illegal to me
XorNot 5y ago
Apple are proposing to do it on your phone. There is no possible way they are not going to do it to everything on your phone. That's a big policy change, since they're circumventing their own E2EE using their super-admin powers to do it. The barrier when uploading to unencrypted services is that you are at least aware you're granting access. This is invading the phone to do it anyway - that's quite different. But more importantly: when flagged this is going to send the results to human reviewers. Which has not been elaborated on: there's no way human reviewers can screen things without being sent a copy of the image in question. Which means, a neural network system is going to randomly send your personal images to human reviewers - and it will, largely, be false positives. Note that Apple are not discussing anything about the reliability of this system: have they run it against a sample set of normal images? How many did it flag? Because, if they give a number, it'll be pretty easy for people to realize that some % multiplied by the number of phones and average number of images per phone, and I'm willing to bet what you get is: the system will false positive at least (and probably more then 1) personal photo per iPhone user, and send it to human reviewers. And that might get people's attention: certainly more then just, unfortunately, us techies. EDIT: And let's talk about these human reviewers: these are not random citizens seeing something and wondering if they should be concerned. The context these people are going to be given is possible child abuse image. This is not a neutral review process - at all.
martimarkov 5y ago
The concern is I bought an iPhone based on the promise of privacy of my device. Now there is a monitoring tool on my phone looking at photos for child porn. I get a false positive and some rando from Apple is now looking at my personal photos. Why? Normalisation of this invasive behaviour is not okay. Apple specifically argued they are not Google and do not invade your privacy.

qubex 5y ago

I’m a nudist. I’m wondering whether we’ll be a persecuted minority (as opposed to merely derided comic relief).

swiley 5y ago
Judging by how Apple has handled the App Store you'll certainly be reported.
monkeynotes 5y ago
What about people with young children? I know my parents have a good number of 35mm shots of me in the bath with my brother and sister when we were babies.
- zimpenfish 5y ago
  > What about people with young children? It's not matching an AI's interpretation of "is this a naked child?" though - it's specifically only matching perceptual hashes against a known database of CSAM percpetual hashes provided by NCMEC which will then be manually reviewed if there's more than a threshold of matches.
  - aaomidi 5y ago
    So we should assume any and all of our photos are probably going to be looked at by some random people? What's stopping said people from leaking photos?
    - zimpenfish 5y ago
      > So we should assume any and all of our photos are probably going to be looked at by some random people? No? Not unless they're matching the perceptual hashes of the CSAM provided by NCMEC and in that case, sorry, my sympathy does not extend that far. If anything, this makes it harder for random Apple people to look at your iCloud photos if moving the matching step down to the phone means iCloud photos are going to be end-to-end encrypted, right? (Which is not a given, of course, just a theory that some people have about why it's moving down to the phone level.)

beebeepka 5y ago

Stuff like this is triggering my whataboutism alarm as not a day goes by without the self proclaimed leaders of the "free world" attacking certain governments on nothing but suspicion and "intelligence". Before 2013, the more cynical ones among us were simply branded conspiracy nuts and crackpots. If anyone believed Apple's lies about privacy, that's on them. The solution is to buy more smart stuff from American companies. They are the only ones that can be trusted

whiddershins 5y ago

Does it matter that they are only doing this for images uploaded to iCloud?

ballenf 5y ago

If we don't own the phone, we have no basis to insist on a search warrant before a search. So moves to make it obvious that Apple really owns the phone and we're just pushing buttons on it, undermines any efforts to establish legal precedents around searches and seizures.

gruez 5y ago
>If we don't own the phone, we have no basis to insist on a search warrant before a search. Thats not how it works. Your house is still protected from search and seizure even if you rent it.

oakfr 5y ago

FWIW Microsoft already does this for photos you store on OneDrive: https://www.theregister.com/2014/08/07/ms_tip_off_child_abuse_arrest/ https://www.theregister.com/2014/08/07/ms_tip_off_child_abus...

oauea 5y ago
That's great but irrelevant. This discussion is about scanning them locally on your device.
- oakfr 5y ago
  Indeed. Thanks for pointing this out. Scanning on device is a different story.
  - deleted 5y ago
    [deleted]
- zimpenfish 5y ago
  Well, it's semi-relevant, isn't it? Because it's only scanning photos that are being uploaded to iCloud Photos - they've just moved the scanning portion down one level. If your photos never go to iCloud, they won't be scanned.
  - cageface 5y ago
    So what's the point of doing this client side then? It seems like they're deliberating paving the way for doing this on all client side material.
    - zimpenfish 5y ago
      > So what's the point of doing this client side then? Not a clue. Maybe they're going to make iCloud Photos fully E2E and this is the only way they can keep the CSAM scanning alive? Or maybe they just want to avoid it arriving on their servers at all (but I don't think anything has said matching CSAM photos will not be uploaded?) Or they just want to save some CPU on the iCloud side by punting that down to the phones themselves? (Or, yeah, this is just the precursor to doing more client-side scanning and the slippery slope cassandras will be proven correct.)

markus_zhang 5y ago

Time to use a dumb phone. I realized recently that I can uninstall most apps on my modern phone and still live without any issue. That's the privilege of immigrating to a foreign country as friends and family now don't expect you to reply as soon as possible.

vegetablepotpie 5y ago

How likely are perceptual hashes to give a false positive? If I take a picture of a tree, how likely is it that a few pixels are going to line up just right in a hashing algorithm and say it might be child porn? How likely is it that law enforcement is going to understand the limitations of this technology? How likely is it that the judicial system will understand the limitations of this technology before issuing a search warrant? I can see law enforcement showing up at my door one day demanding to have a look around, and I would have no idea why they’re there, but they’ll want to look through all my personal belongings. Worse yet, I might come home from work one day, see my windows broken, see my place has been ransacked and my computers are missing. I would call the police to report a burglary only to hear that I’m under investigation and they need me to give them the key to decrypt my hard drives. I feel like I need to do some risk analysis on my digital life. I need my phone for 2FA apps for investment accounts. If I use a dumb phone, I’m stuck with SMS, which is easy to hack. But if we use iPhones, there is a small but costly risk of a completely erroneous investigation. How do you balance these without knowing the probability and impact of the later?

markus_zhang 5y ago
EDIT: OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. -------- At least it's a lot more possible to mis classify a photo than a sms message. Plus as I mentioned in another post, what if you have a young kid and you happen to shoot a photo when he or she is not fully dressed?
- hermannj314 5y ago
  Why would that photo match a list of hashes of known CSAM though? This technology at Apple is very much about matching hashes.
  - markus_zhang 5y ago
    Thanks, original post edited.

ByteWelder 5y ago

A better title would be: Apple to scan iPhones for banned images. SwiftOnSecurity: > Just to state: Apple's scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn't know what a child is. (from https://twitter.com/SwiftOnSecurity/status/1423383256003747840 https://twitter.com/SwiftOnSecurity/status/14233832560037478...)

hef19898 5y ago
Nice. Well, Apple had me consider a switch from Pixel phones when they prioritized privacy over Facebook. Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google. And they were that close to become an acceptable big tech co regarding user privacy...
- zimpenfish 5y ago
  > Since I don't want anyone scanning and deeming my pictures "acceptable" I just stick with Google. Don't Google already do this for images you store in their cloud?
  - rvz 5y ago
    yes. [0] https://www.theverge.com/2014/8/5/5970141/how-google-scans-your-gmail-for-child-porn https://www.theverge.com/2014/8/5/5970141/how-google-scans-y... [1] https://torrentfreak.com/google-drive-uses-hash-matching-detect-pirated-content/ https://torrentfreak.com/google-drive-uses-hash-matching-det...
  - temeritatis 5y ago
    Sure, but so does icloud and onedrive. And while i don't like that either, at least one can opt out of it. The only way to realistically opt out of this new system is to not use an iOS device. And i'm just wondering how long until they want to scan your mac as well.

markus_zhang 5y ago

EDIT: OK I didn't read the article, that's my fault. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud. ---------- I want to add that this may be a serious issue for people with young kids. Imagine you shoot a video/photo of your kid with little dressed on. Now what happens? Are we supposed to fully dress kids because we have no idea what Apple will do after the scan?

tunnuz 5y ago
This.
- Jowsey 5y ago
  Not this. Read the damn article, people. It clearly says it's matching pre-determined hashes, nothing to do with AI. You're making this worse than it has to be.
  - markus_zhang 5y ago
    OK I didn't read the article, that's my fault and thanks for the clarification. But the original point stands: I don't want Apple run more scans on my phone. They can scan on Cloud, but not on my phone before it's loaded into Cloud.
tpush 5y ago
It checks images against an on-device database of known CSAM, your photos should be fine.
- markus_zhang 5y ago
  OK. Another argument: what if a virus download the pic for you?
  - clifdweller 5y ago

solarkraft 5y ago

Looks like that concludes my otherwise very nice time in the Apple ecosystem. Luckily Linux is coming along well both on phones and the new Macs.

bjarneh 5y ago

Slippery slope...

gerhardi 5y ago

So would this make a good vector for black mailing? Also I guess many parents might have a lot of pictures of their small children playing around naked.. at least where I live.

cheese_van 5y ago

Powerful corporations aligning interests with law enforcement by assuming certain LE roles seems a recipe for corporate dystopian oligarchy. This opens, for Apple, a catalog of answers for "How can we make a buck," and none of the answers will be friendly to privacy or freedom.

2OEH8eoCRo0 5y ago

Let me get this straight- the iPhone will now scan your iCloud account and compute hashes for images. These hashes are then compared against a known database of child abuse images. When a certain threshold of "hits" is reached then steps are taken to notify authorities? There seems to be little chance of false positives here. I keep seeing references to machine learning algos and that's not what they are doing at all. I do still lament losing yet more control though. A device that I buy should not be working for somebody else, for better or worse. CSAM is not new and is already in widespread use. The difference here just seems to be that your device itself will sell you out. Edit: Follow up question- They run this on images before they are uploaded to iCloud. Why has Apple chosen to do this on-device rather than just run it in iCloud? They say for privacy reasons: 'Apple’s method of detecting known CSAM is designed with user privacy in mind.'

whitepaint 5y ago
Isn't this very very silly? You just need to change one pixel and the hash will be different.
- deleted 5y ago
  [deleted]
- krapp 5y ago
  No, perceptual hashes don't work that way. It's a comparison of similarity to a target, not just making an MD5 hash of an image. Here's Apple's documentation on it : https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - whitepaint 5y ago
    Oh nice, didn't know about this.
  - 2OEH8eoCRo0 5y ago
    Very helpful. Thanks! Will give this a read.

syntaxfree 5y ago

So wait, I have pictures of my baby son’s poop to send the pediatrician — which show what will eventually be his naughty bits, as he becomes sexual (in due time, years from now) I actually don’t have many nudeish pics of him because it’s very cold right now, but we take dozens and dozens of daily pictures so — but in the summer… when maybe he’s going to the kiddie pool grandpa has…

merricksb 5y ago

Earlier discussions: Apple's plan to “think different” about encryption opens a backdoor to your life – https://news.ycombinator.com/item?id=28079171 https://news.ycombinator.com/item?id=28079171 – Aug 2021 (748 comments) Expanded Protections for Children - https://news.ycombinator.com/item?id=28078115 https://news.ycombinator.com/item?id=28078115 - Aug 2021 (353 comments) Apple plans to scan US iPhones for child abuse imagery - https://news.ycombinator.com/item?id=28075021 https://news.ycombinator.com/item?id=28075021 - Aug 2021 (368 comments) Apple enabling client-side CSAM scanning on iPhone tomorrow - https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741 - Aug 2021 (716 comments)

deleted 5y ago

[deleted]

White_Wolf 5y ago

what stops apple from checking if you have any content they don't like of like non-pc memes and such and flag you in some databse for futher monitoring/enquiry by pc police?

clifdweller 5y ago
that is the big concern that in future you share pepe frog meme or in say russia a pride flag and it gets marked as terrorist image and they report you to _____ or block you from posting on ______

me_me_me 5y ago

I wonder if this is secretly brilliant stunt. Blast on every channel message that your product will actively look for children porn store on it. Stir great debate. And then quietly drop the proposed feature. Your product will be viewed as unsafe to use by the pedophiles maybe even whole apple ecosystem will.

jadecolour 5y ago

I guess it's just a disguised protest against the government by Apple, that they have to give up their stance on privacy and make the world know. Yes, children is so precious and helpless. I think in order to protect them you all have to give up your liberty, so to protect the future of human race.