27 ms·

Apple says photos in iCloud will be checked by child abuse detection system

461 points by shivbhatt 5y ago

drglitch 5y ago
Who needs SWATing when you can send a CP pic (either real or with hash collision as per the thread few days ago) from a virtual overseas number/service and get FBI van to show up as well? What about injecting code into a public website to download same pic into local browser cache without user’s knowledge? The simplicity of the attack vectors here that would trigger the “manual” investigation is just dumbfounding and ripe for abuse/misuse.
- bpoyner 5y ago
  You could completely wreck somebody's life with this. SWATing will look trivial in comparison.
  - scbrg 5y ago
    To be fair, SWATing kills people. Death is generally considered a non trivial and also life wrecking event.
    - headmelted 5y ago
      Honestly I'd rather get shot dead by a SWAT team than implicated for something as atrocious as what this tool is looking for. I imagine many people with a family would feel the same way. It's an abomination that will destroy innocent people. The engineers behind this no doubt think it's fool-proof because they believe they're leagues smarter than any of those pesky naysayers ("hey, we're Apple"). If we've learned anything about Apple this year (as if we needed the reminder) is that their software is nowhere close to as flawless as they seem to think it is.
  - headmelted 5y ago
    This. Combined with the unpatched remote-root-via-phone-number disclosed in the Pegasus leak this boils down to a single-click "destroy this person's life" tool.
  - donkeyd 5y ago
    If you assume that cops will just arrest people without doing any further research... Then yeah.
    - shakna 5y ago
      > If you assume that cops will just arrest people without doing any further research... Then yeah. Like when they arrested & charged someone for a poor facial recognition match that never had a hope of passing human review? [0] Just glancing at the original photo would have stopped that. Or checking his rock-solid alibi. Neither of those things happened. [0] https://www.wired.com/story/flawed-facial-recognition-system-sent-man-jail/ https://www.wired.com/story/flawed-facial-recognition-system...
    - bpoyner 5y ago
      Oh, the police will get a search warrant, and find exactly what they were told would be on your device. The police aren't in the business of discovering your innocence. It's then up to you and your lawyer to prove you didn't put it on your device. Meanwhile your life will fall apart as you get fired, your wife divorces you, you lose all custody of your kids, etc.
  - dzhiurgis 5y ago
    You can already do the same. Send message then call the cops. Just because it's auto-detected now doesn't mean it wasn't possible before.
- hughrr 5y ago
  This is why I just noped the fuck out of the Apple ecosystem. I won’t support anything which relies on opaque blacklisting to ruin lives. In this example as well on iCloud shared galleries you can upload to other people’s ones you have been invited to. What could possibly go wrong?
  - jstx1 5y ago
    Maybe wait to see how it's implemented and how it works first? I really think that the HN crowd is having a giant knee-jerk reaction to all of this.
    - bilekas 5y ago
      Its harder to take back policies like this than it is to object and get them stopped initially. Also people have a habit of 'forgetting' about it later. Until stories of how it is misused are found. And then it's another attack vector we need to be conscious of.
      - ElFitz 5y ago
        And that’s how France still has VAT & revenue taxes. Revenue tax? Have to pay for that expensive WWI war effort, you understand? For all the good it did. Same with the VAT. Have to rebuild after WWII, you understand. We also have an "Exceptional and Temporary Contribution" (CET), recently renamed to "Technical Equilibrium Contribution" (still CET. Smart one, that one). A funny one, for a change? When the Germans invaded in WWII, they changed France's timezone to theirs. After the war, we still called it "the German time". There were talks of going back for a few years… Guess who still has noon at 2pm in the summer, decades later? Change, no matter how ridiculously small or sensical, even when nobody benefits from the status quo (ie the damn timezone) is horrendously difficult. Thus one should always assume that once it’s here, whatever "it" is, it’s here to stay.
        candiodari 5y ago
        There is still one constant: how the state system cares for victims of child abuse is still the same as in WW2. https://www.kansascity.com/news/special-reports/article238206754.html https://www.kansascity.com/news/special-reports/article23820... You would think money would go into the "backend": caring for kids where the state is responsible for everything BEFORE more money goes into the frontend: finding more kids to throw into the hellhole that is child services. Without the "backend" being in order and working well, raising well-educated, stable kids, the frontend is completely immoral. "Saving" kids from abuse, only to throw them into a slightly different kind of abuse ... if any person did that (e.g. a guy marrying a woman (or I guess vice-versa) with that resulting in that person abusing their new spouse's kids) would be considered a despicable crime. Somehow child services, who do the exact same thing (and they use violence to do it) is not a despicable crime. Somehow just because the state does it, makes such things all a-okay. But frankly this is merely the hole in the justification, all this should merely tell you one thing: any government that doesn't work hard to fix the child services backend does not have children's interests at heart when making these sorts of laws (and mostly they're making budget cuts in the backend, of course). Because fundamentally these laws throw children into the child services system. THAT is the real effect these efforts have on the actual children behind this. THAT is what is meant by "saving kids". And if that system is full of abuse, how is that any better than what paedophiles do? It's not. Which means the state is not attempting to help abused or disadvantaged children. In fact, they're doing the opposite.
- jacquesm 5y ago
  No code required. <img width=0 height=0> would do the job.
  - oleganza 5y ago
    That's fine™. You are just going to redirect blame on the original source, provided you got enough Apple Cash on balance to pay the lawyers and stay out of jail while sorting this out.
  - mimsee 5y ago
    I wonder how long it takes until they add a feature to Safari to scan all the <img> <video> <canvas> elements for possibly illegal content. Would be very convenient considering Safari is the only browser engine on iOS.
  - paulcole 5y ago
    How is that going to get the image into your iCloud photo storage?
    - jacquesm 5y ago
      It doesn't, but it does get the image into your browser cache and onto your machine.
      - paulcole 5y ago
        So what does that do in the context of this conversation about Apple and iCloud?
    - schoolornot 5y ago
      Content-Disposition: attachment; hit the wrong button, done. It's in your iCloud/Downloads folder.
  - madmoose 5y ago
    > <img width=0 height=0> would do the job. No, that's not how the Apple's system works.
- mimsee 5y ago
  Yes. This reminds me of when typing or receiving certain text would make an iPhone crash. But now having your account deleted makes it a feature. For example Whatsapp automatically downloads media to the camera roll which then get uploaded to iCloud. Of course that can be turned off prior, but this is like what happens with backing up. People want to backup, but don't invest the time in it. That's until it's too late, they lost their data and now want their stuff back.
  - laurent92 5y ago
    Backup is a good point: - Apple: “Backup your phone to iCloud, it will be safe there.” - 5 minutes later: “We’ve wiped your account because of a photos of (porn actor here) which is not CP but technically minor at the time she filmed.” - “Also we’ve wiped your iPhone because we couldn’t knowingly let you keep that. Good luck contacting your parents, we’ve deleted your contacts. Good luck! PS: We’ve reported you to the police.” - Also you can’t connect to your iMac now.
    - annamargot 5y ago
      Or photos of your own children. We have a Tumblr set up for family to view pics of the kids. Several photos and videos of our kids when they were under 2 were taken down either temporarily or permanently by their CP algo. These were a pic or video of kids in the bath or without a shirt. In none of them could you see bum or bits. Just a semi naked baby. Algorithms like this get things wrong all the time
      - tpush 5y ago
        This is not the kind of algorithm that Apple is be using. That one only scans for already known CSAM in NCMEC's database.
        shakna 5y ago
        Which may contain the hashes of their photos, because they've been taken down in the past, which means they probably have been added to certain blacklists that may have been integrated into the blackbox of NCMEC's database.
- simondotau 5y ago
  None of those attacks would work against the system as described by Apple. The only photos scanned are items in your photo library prior to upload to iCloud. Your browser cache is not scanned. Hash collisions would fail human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.
  - soziawa 5y ago
    > Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM. The whitepapers provided by Apple do not say what the human reviews consists of. They could just look at the hashes to make sure there isn‘t a bug in their system.
    - simondotau 5y ago
      > The whitepapers provided by Apple do not say what the human reviews consists of. At minimum what we know is that each flagged image generates a "safety voucher" which consists of metadata, plus a low-resolution greyscale version of the image. The human review process involves viewing the metadata and thumbnail content enclosed in each safety voucher which cumulatively caused that account to be flagged.
      - foobar33333 5y ago
        A human at Apple likely doesn't get access to anything. I assume it would be part of the police group under strict restrictions checking these.
        simondotau 5y ago
        The data is not sent to a "police group", it is sent to NCMEC. From Apple's FAQ: Will CSAM detection in iCloud Photos falsely flag innocent people to law enforcement? No. The system is designed to be very accurate, and the likelihood that the system would incorrectly flag any given account is less than one in one trillion per year. In addition, any time an account is flagged by the system, Apple conducts human review before making a report to NCMEC. As a result, system errors or attacks will not result in innocent people being reported to NCMEC.
- sschueller 5y ago
  My public Wifi captivity portal...
- 2OEH8eoCRo0 5y ago
  Don't worry. I'm sure the police will believe you and help you out. /s What you've described is pretty much the scariest thing I can imagine as far as computer crime goes.
- ratww 5y ago
  I remember WhatsApp used to save each received image to the iCloud Photo Album. I remember one day going to my album and seeing several memes and pics I had received but never saved. Having 3rd party apps that have access to the photo album being able to do that makes it a bit risky to have iCloud.
  - EtienneK 5y ago
    WhatsApp was my first thought as well. Any app that automatically saves photos to iCloud without user interaction is a huge risk.
- jbverschoor 5y ago
  It's a good method of protecting important documents. Simply add some stamps on top of all documents in case someone steals them
- sylens 5y ago
  You don't even need to inject code into a public website. There have been no shortage of zero-click exploits for iMessage
- modernerd 5y ago
  The reported response from Apple offers little reassurance: > The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material. But they said they expected any such attacks to be very rare and that in any case a review would then look for other signs of criminal hacking. What triggers them to look for signs of criminal hacking? Does every manual review process involve such checks? Are they searching device backups for indicators of compromise [IoC]? What if there's no device backup or device image to scan? What if the scan fails to notice IoC? What if the device was compromised after the last backup? What if the device was compromised via physical access? What if the device isn't compromised and the material was pushed maliciously or via drive-by download? It's dangerous to assume that all material on a network-connected device arrived with the consent of the user when it can accept incoming messages from strangers, trick people into downloading files, or be compromised without your knowledge. “That isn't mine” is going to be a tough defence if you can't even take measures to log where content came from. Client-side scanning seems to amplify this issue (which could still happen with cloud storage) because at least cloud storage doesn't generally ship with or integrate deeply with messaging apps, social media, a web browser, QR codes, App Clip Codes[1] etc. The impact might be fairly low right now with the current proposal (images would have to be uploaded to iCloud, so cached browser images don't get scanned as far as we know), but the existence of the non-consensual scan in the first place is worrying, because it means such attacks are only a policy change away. [1] : https://developer.apple.com/design/human-interface-guidelines/app-clips/overview/app-clip-codes/ https://developer.apple.com/design/human-interface-guideline...
  - GeekyBear 5y ago
    > The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material. Since Google has been scanning your account for kiddie porn for the past decade, wouldn't this apply equally to Google accounts? >a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-led-to-a-mans-arrest-for-child-porn-was-not-a-privacy-violation/ https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le... All people have to do is email you kiddie porn and Google will have you arrested?
- quietbritishjim 5y ago
  Something slightly different but very related happened to a senior police officer in the UK. She got sent a WhatsApp message by her sister containing a horrific CP act. It was captioned with a message asking people to circulate it to identify the adult in it, and probably those who sent it around (including the sister) were acting in good faith, but actually it was still illegal to send or even possess it. No doubt the originator of the caption was a deliberate troll. She was found guilty of "possessing an indecent image of a child". [1] She tried to argue that she hadn't noticed the message, but it's not surprising that wasn't believed given that she had immediately replied to her sister saying "please call". She was sentenced to 200 hours community service, and originally sacked from her job but recently reinstated after appealing. [2] It seems that she wasn't immediately in trouble when she received the message ... so long as she had immediately reported her own sister for distributing it, even though it's clear that she hadn't deliberately done anything wrong. (In fact the sister had contacted her to ask what she should do about it. Probably her answer was "don't have already sent it me!") [1] https://www.bbc.co.uk/news/uk-england-london-50476166 https://www.bbc.co.uk/news/uk-england-london-50476166 [2] https://www.bbc.co.uk/news/uk-england-london-57501764 https://www.bbc.co.uk/news/uk-england-london-57501764
  - zimpenfish 5y ago
    > a senior police officer in the UK To be fair, this is partially because the laws in the UK are, I think, fairly bonkers strict about CSAM - mere possession, whether you've looked at it or not, whether you downloaded it or not, whether you even know it's there or not, etc., is counted as criminal.
    - zionic 5y ago
      The US is the same.
      - wutbrodo 5y ago
        I believe this is incorrect. > At the same time, because of the First Amendment, child pornography offenses are not "strict liability" crimes like statutory rape: in order to convict a defendant, the government must prove that the defendant knew the material involved the actual abuse of a child https://www.zmolaw.com/child-pornography-faqs# https://www.zmolaw.com/child-pornography-faqs# I've found similar claims on the websites of a few law offices. For some reason, the official DoJ materials are pretty cagey on the topic.
- nicce 5y ago
  > Who needs SWATing when you can send a CP pic (either real or with hash collision as per the thread few days ago) from a virtual overseas number/service and get FBI van to show up as well? You are talking like collisions are trivial to make. I bet they have had a deep conversations in this area. At first, you would need a real hash to even try (which are hidden). Secondly, to get real material it means that it must be in their database to trigger anything. This tells a lot from sender already, and is worth to tell for police. It is quite easy to prove that someone just send it to you. And one photo is not triggering anything. Besides, sender must know that those photos must go automatically into the cloud to mean anything. > What about injecting code into a public website to download same pic into local browser cache without user’s knowledge? At least US legistlation is precise that user must willingly obtain/download CSAM material, and it must be proved. So this is not harmful for the user in the end. A lot of speculation, but does not really lead for coencequences. Almost every system can be tried to be abused, but does it really mean something, is different story.
  - vineyardmike 5y ago
    > At least US legistlation > does not really lead for coencequences Except that a trial, even with an innocent verdict will SUCK and have terrible news stories about you and poison any google search for you with CSAM stories
  - themaninthedark 5y ago
    Step 1: Get copies of pictures of targets kid in bath from phone/SNS Step 2: Manipulate pictures so that hash collides with CSAM Step 3: Get pictures back on targets phone so they get scanned. I don't have the skills or understanding of how the hashes are created but would this be possible? >At first, you would need a real hash to even try (which are hidden). How are the hashes hidden? It looks like they are shared: https://www.thorn.org/reporting-child-sexual-abuse-content-shared-hash/ https://www.thorn.org/reporting-child-sexual-abuse-content-s...
    - nicce 5y ago
      > How are the hashes hidden? It looks like they are shared: https://www.thorn.org/reporting-child-sexual-abuse-content-s https://www.thorn.org/reporting-child-sexual-abuse-content-s.. These hashes are not generated by Apple and are not valid. (Must be generated by their new system) They are probably very strictly guarded. They will be stored on every iOS from 15 version, somehow securely. This must limit the support of older iPhones.
- robertoandred 5y ago
  Just because you assume attack vectors are simple doesn't mean they are. First of all, why would Apple forward a report about something that isn't CSAM to the NCMEC?
988747 5y ago
And what if I have iCloud storage turned off? Will this local scanning still be applied?
- bkishan 5y ago
  Nope, only for photos in iCloud and destined for upload to iCloud.
  - roody15 5y ago
    yeup wink wink… the scanner will still be there … but Apple promises not to use it for anything else and only with photos going to upload. Wheww that makes me feel better
  - echelon 5y ago
    Apple isn't trustworthy. I bet this stance changes in under five years.
    - madmoose 5y ago
      > Apple isn't trustworthy. Why not? They've told you in detail what they're going to do.
      - xvector 5y ago
        > Why not backtracking on promises after buying from them
      - SheinhardtWigCo 5y ago
        Lando Calrissian: That wasn't part of the deal! You said the wookiee and Leia would stay under my protection! Darth Vader: I am altering the deal. Pray I don't alter it any further. [departs in an elevator]
  - sneak 5y ago
    There's no point in clientside scanning, then. The photos in iCloud are already not e2e and are completely readable by Apple (and are regularly turned over to the USG without probable cause or a search warrant).
verytrivial 5y ago
I've read this whole situation as a signal to China and other authoritarian regimes that Apple has finally seen diminishing returns from the "Apple is more secure" angle and is now looking elsewhere for growth. It's just business.
- SquishyPanda23 5y ago
  I think there is more going on here. I think what we're seeing is Apple betting on using cryptography as part of the product design phase. Apple devices already do weird things like wake up to announce their physical location so that users can find their devices. The thought of a powered down or suspended laptop waking up to announce its location isn't something I particularly want, but Apple users seem to like it. Anyone who has spent any time on spaces that are strongly encrypted and focused on privacy know how quickly they become havens for the sort of material that Apple doesn't want associated with its brand. How many "Apple protects child predator" news stories do you think Apple can withstand while still remaining a luxury brand? Apples goal here is to have the reputation for end-to-end encryption and privacy while simultaneously not being seen as a phone for child predators. They don't have a lot of options if they want to thread that needle. I've thought about this space quite a bit, and all options suck. Client side scanning is really the only choice with reasonable tradeoffs. The other option is scanning encrypted photos on cloud using secure enclaves to do the scanning. My guess is that when the tech makes that possible Apple will move in that direction. I agree that this isn't the best for privacy nuts like me. But the iPhone isn't a blackphone, it's a luxury handbag. The phone isn't for privacy nerds, the privacy is there to make other mobile OS's look cheap and tacky.
  - zionic 5y ago
    This is needless Apple apologetics. They deserve to be raked over the coals for this, there's no world where their current design is a "good" or "right" one. Child abuse is a serious problem, but building a surveillance panopticon is not an acceptable solution to it. Better investment in education, health care, and reporting hotlines are the way forward to stop this issue at its source.
Odenwaelder 5y ago
Maybe someone can comment on this: Does Google scan the cloud fotos of its users for CP? Have we seen an uptick of false positives/SWATings since they do that? Apple is - rightfully and understandably IMO - criticized for their plans, but does anyone know how Google handles this?
- soziawa 5y ago
  There is some previous discussion about this regarding Facebook here: https://news.ycombinator.com/item?id=28119372 https://news.ycombinator.com/item?id=28119372 The false positive rate does not look great.
- sneak 5y ago
  Google and FB both scan storage for several different types of contraband, and also have triggers and thresholds for things that use too much bandwidth (eg pirated software download links that are shared widely et c).
- bengale 5y ago
  Yes they do. The reason apple has done this is because they lagged behind other providers considerably in detecting this sort of content. Facebook for example are reporting millions per year compared to a few hundred for Apple. Instead of scanning you whole library they came up with a way to do it on device, which is the main difference between other services. If you don't enable iCloud photo storage the system can't work at all. Very interesting stuff in their technical explanation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - zionic 5y ago
    -Apple "lagged behind" because it built private and secure services it could not monitor by design. This is a feature, not a bug. -Facebook's reporting overwhelmingly flags burner accounts signed up via tor etc, only absolute idiots would post actual CP on their real name account on facebook. -Apples solution is highly invasive and dangerous, and your statement about "only running with iCloud upload" is false. It took less than a week for Apple to announce that they will open these APIs to 3rd party apps.
new299 5y ago
Apple have already been doing this for sometime: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... This is about the new PSI system: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ In which photos are scanned on the users device. This appears to be a report of a new press conference after the initial announcement? Does anyone have a transcript of this press conference?
techbio 5y ago
Apple is smart enough to determine what's in an photograph, but if I paste text into a sentence in iOS I still need to manually add spaces and format punctuation. I will never forget the time iTunes deleted my music library, or it's inability to deduplicate identical songs. Power.
- mabedan 5y ago
  do this day, if you switch the country of your apple account, you will lose your entire music library.
  - sneak 5y ago
    Cloud music libraries aren't yours to begin with, even when working right. Torrent the actual files.
- DavideNL 5y ago
  Even worse, iTunes deleted my Music and all my precious playlists, and because it was stored in the cloud it wasn’t included in my Macbook backups… I was so pissed of, i’ve avoided iTunes & Apple Music ever since.
- insickness 5y ago
  I hate hate hate iTunes. My biggest gripe with iPhone has always been that I don't have direct access to my OS files. I have to spend an inordinate amount of time 'syncing' my device instead of just copying a few tracks to it.
  - robomartin 5y ago
    > I hate hate hate iTunes Yup. Same here. This is why I don't have my large collection of CD's in my phone and just use Pandora. Going back to iPhone 3 days, iTunes did not allow me to import my CD's into my device and play them as entire CD's. If I want to listen to Mozart's "Eine kleine Nachtmusik" or Pink Floyd's "The Wall", it's a nightmare. iTunes is song-based, not album based. Well, the above, and many others, are works you pretty much listen to in order as recorded. In some cases (The Wall, Brandenburg Concertos, etc.) the works span multiple CD's. I stopped using iTunes and storing music on my iPhone because of this. I don't enjoy music the way Apple seems to think you should. I have no clue if they fixed this since iPhone 3 days. I would not be surprised if they have not. In sharp contrast to this, I have not problem playing single or multi-CD works as intended using Windows Media Player on my desktop, where I have my entire CD collection stored. This, for me, is the single reason I would instantly jump into a Windows phone if Microsoft got their heads out of their asses, committed to doing a good job and integrated a phone experience with the desktop. They would have to regain my trust, but as a life-long user of both Apple and MS desktop products, I would absolutely welcome a better phone experience than Apple has delivered over the years. I really want to abandon iPhone and go to a good Windows phone, but MS does not seem interested in creating that opportunity. Oh, yes, and to address iCloud, back in the early days it managed to delete not only whatever I had on iTunes (which I own on CD's so I don't care) but all of my contacts. Thankfully I had my contacts stored in my prior phone (I think it was a Blackberry). After disconnecting from iCloud I entered them manually and never again enabled iCloud all the way up to my current iPhone X.
deleted 5y ago
[deleted]
endisneigh 5y ago
Which cloud provider doesn’t do this and would be unable to do this at any point in the future?
- TroisM 5y ago
  Any of them as long as you encrypt before you upload.
- lazzurs 5y ago
  Proton. https://protonmail.com/blog/protondrive-security/ https://protonmail.com/blog/protondrive-security/
  - foobiekr 5y ago
    The first stage of their receive pipeline has access to the plaintext. Ditto the send pipeline.
salmo 5y ago
Wow. This could be messed up for attorneys, DCS, social workers, etc. They allude more to child pornography, but I hope it doesn't extend to physical abuse. Those photos are usually taken on phones by spouses, doctors, schools, etc. to be passed to the above on their phone for evidence for a DNN or similar case. Glad my kids have aged out of baby bath photos. And those poor people who I know are going to have to provide an auditing safeguard. I hope they take care of their mental health.
- judge2020 5y ago
  In every thread about this, someone makes this same false assumption: no, Apple is not scanning for naked children or children in pain. It's generating a hash to be compared against hashes of NCMEC-verified CSAM pictures (and while some HN commenters claim the DB contains non-CSAM, that has not been verified nor ever reported on by a news publication) and it does indeed only scan photos destined for iCloud Photos (which I theorize is the only part keeping this system legal[0]). 0: https://news.ycombinator.com/item?id=28112982 https://news.ycombinator.com/item?id=28112982
  - zionic 5y ago
    >that has not been verified nor ever reported on by a news publication) How could it be? The list is literally property of the secret police, you can't know what's on it. No one can audit it except the police themselves.
    - judge2020 5y ago
      Auditing it would mean looking at CSAM, which is illegal. If it was widespread, I would expect at least one of the manual reviewers that are able to legally view the CSAM would contact major publications (under the promise of staying anonymous) and whistleblow on this issue. : To be clear, I'd expect whistleblowing if these manual reviewers were tasked to 'accept' CSAM submissions that aren't CSAM.
      - zionic 5y ago
beshrkayali 5y ago
I stopped using stock Android and went back to iPhones because I thought Apple cares more about privacy than Google does. Not exactly correct in all cases I know (ie. they both suck in terms of privacy), but it seemed like Apple users are buying in, so it might work. Now I think my next mobile OS is going to be GrapheneOS. Like others have mentioned, this is as big of a warning as anyone's going to get to get out of that locked-in ecosystem. On that note, the outrage is kind of useless if you don't skip buying the next iPhone. You should fully own what you fully pay for.
- Maxburn 5y ago
  Hail to Graphene being the official No Agenda Phone https://noagendaphone.com https://noagendaphone.com
  - tbihl 5y ago
    Why disable 5G? I was following everything until that point, but I'm not sure the reason for that choice.
    - dimitrios1 5y ago
      I am curious as well. My guesses are a. 5G allows for metadata collection points b. 5G chips are much more expensive because of patents
      - Maxburn 5y ago
        Me too. A; maybe, I'm not sure what extra that does above LTE. B; I guess you could get a different phone that doesn't have it, for a little while longer.
    - Maxburn 5y ago
      I'm not sure myself, I just know they push Graphene pretty heavily on the No Agenda show.
    - da_big_ghey 5y ago
hkai 5y ago
So... Any predictions for how long till this is used to target opposition?
hkai 5y ago
I am totally making an npm package that secretly generates a CP image during the postinstall script. Developers created this and developers must suffer from this.
- zionic 5y ago
  I know this is a joke post, but I think in all seriousness it's going to take a crisis like this to get these laws/society changed. "Thousands of developers swept up in CP ring!" that later turns out to be malware planting CP would go a long way towards fixing this issue. I really am surprised nobody has made a worm that's sole function is to hit every FBI honeypot in existence and archive it to hidden folders just to prove a point.
EGreg 5y ago
I find a marked difference on HN about the attitude towards end-to-end encryption, and anonymous transactions in cryptocurrencies. The former can enable the latter. And so much more. Organizing sex trafficking, terrorism and so forth. Nevermind the copyright protection stuff. The latter can enable tax evasion, money laundering and financing unsavory activities. States don’t want people to be able to do that. Yet many on HN applaud attempts to doxx everyone and every transaction in crypto, calling it a scam/for criminals, while at the same time decry any attempts to lessen encryption, however subtle or careful, of personal files and communication. What is a consistent position on both these topics, given that there are dangers on both sides of the argument? I tried to present the core issue here: https://news.ycombinator.com/item?id=28117289 https://news.ycombinator.com/item?id=28117289
CTDOCodebases 5y ago
So Apple wants to use my electricity/computation instead of doing it on their own servers. Disregarding the privacy issues that’s pretty scummy IMHO.
- GeekyBear 5y ago
  Google and the rest do the same thing, but on their own servers. You want Google, for example, to hold false positive data on their servers forever where it can be subpoenaed and misused? >Innocent man, 23, sues Arizona police for $1.5million after being arrested for murder and jailed for six days when Google's GPS tracker wrongly placed him at the scene of the 2018 crime https://www.dailymail.co.uk/news/article-7897319/Police-arrested-innocent-man-murder-using-Google-location-data.html https://www.dailymail.co.uk/news/article-7897319/Police-arre...
  - CTDOCodebases 5y ago
    What’s not to say the logs of the scans being performed on my device will be uploaded and stored off my device forever anyway? The point I was trying to make was privacy reasons aside their motivation of doing it on the users device is scummy. Why don’t they mine Bitcoin on my iPhone while they are at it?
    - GeekyBear 5y ago
      If there is a false positive, I don't want that fact to ever leave my phone, instead of residing on Google's servers forever, where it can be subpoenaed and misused. Apple's approach here is far superior from a privacy standpoint. >1. Only if you're uploading files are the files matched. 2. Only if the matches are very close are they considered matches. 3. Only if you have multiple very close matches is Apple able to decrypt the low-res versions of the images themselves. 4. Only if a human reviewer discovers any of the decrypted low-res images to be illegal content is any of your information shared with anyone else. https://news.ycombinator.com/item?id=28120598 https://news.ycombinator.com/item?id=28120598
      - 5y ago
ddtaylor 5y ago
I'm confused. I was under the assumption that they were only going to do client side detection, is this article claiming they will be running the scans in the cloud as well?
- bengale 5y ago
  No, they are putting this in place to avoid doing this in the cloud against your whole library and having access to all of the information that generates. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
squarefoot 5y ago
> "But they said they expected any such attacks to be very rare ..." Well, ransomware has been rare almost forever, then suddenly became the norm. > "and that in any case a review would then look for other signs of criminal hacking." Good luck finding a malicious app that downloads child porn from an encrypted remote server, plants it in the target device, sends "by mistake" an example to social media using the owner credentials, then deletes itself. This is crazy. Child porn traffickers will find a way to circumvent this while it would offer governments just another weapon against people they don't like. Also they completely ignore that we're talking about child porn; if someone is wrongly linked with the subject for just one second by the media, no matter how many times the news is being rectified afterwards, his life may be ruined forever. It's not like being accused of avoiding taxes or theft; any mental association with things like child porn or rape is not going away easily. Any technology that could be (ab)used to plant evidence in such cases would be the ultimate weapon to destroy individuals without actually killing them. Better not to have it than to risk that it ends in the wrong hands.
onepunchedman 5y ago
Can someone please explain to me how this comparison would work? It seems so trivial to alter any image containing CP slightly such that its hash doesn't compare anymore?
- sneak 5y ago
  Perceptual hashes are not related to byte-level hashes.
  - onepunchedman 5y ago
    Yeh I must have ignored the "perceptual" part when I read it over
- dilap 5y ago
  Confusingly, it's a completely different use of the word "hash" See here: https://en.wikipedia.org/wiki/Perceptual_hashing https://en.wikipedia.org/wiki/Perceptual_hashing The goal of a perceptual hash is to generate a number that will be the same for all "similar" looking images. Think like what Shazam does, but in the visual domain.
  - onepunchedman 5y ago
    Thanks for the response, I was super confused by this part!
- bengale 5y ago
  https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - onepunchedman 5y ago
    Thanks for this! :)
3pt14159 5y ago
I had a highly upvoted comment about this whole thing a couple of days ago: https://news.ycombinator.com/item?id=28069528 https://news.ycombinator.com/item?id=28069528 I have come to one more nuance about the viewpoint. If people are spreading CP by signing into the same Apple account from multiple devices and using iCloud to automatically share the photos, I think that's a different situation than a single person signing into one computer and one phone that are mostly used on the same networks together. Not that I've really changed my view that I wrote before, just there is a bit of grey here.
- ericwooley 5y ago
  A) that would be clever, and if the bad guys are that clever, they will easily find another way to share their stuff. B) apple could simply make a policy that your Icloud will trigger a scan / review after x devices are logged into it within a certain amount of time, in order to make sure you aren't using Icloud as a distribution platform for something. This whole scanning locally on device wouldn't be necessary.
tomasreimers 5y ago
If you haven't seen it already, Alex Stamos (former head of Security for FB) has a really incredible Twitter thread on this: https://twitter.com/alexstamos/status/1424054568275439617 https://twitter.com/alexstamos/status/1424054568275439617
- jiggawatts 5y ago
  "Something went wrong. Try reloading."
- zionic 5y ago
  Wow, he's actually suggesting Apple should build a "trust and safety" team for reporting E2E Encrypted messages containing "hate speech" to directly? Sometimes I forget how messed up Twitter is.
  - fsociety 5y ago
    Yes because it’s a mechanism to prevent actual abuse from occurring on their platform and to report offenders directly. When one of the participants of an E2EE conversation reports the convo then the messages would be sent up in a way for Trust and Safety to read the messages and report to the authorities. What is messed up about that? The method of reporting is in the hands of the user, not an ML algorithm. The ML algorithm would prompt the kid to stop and think about what is happening, before actual abuse occurs.. I assure you Stamos is speaking from a place of experience, in having to prevent these kind of things.
    - zionic 5y ago
      My $1000 hardware is my property, not “Apple’s platform”. Also, messaging is a protocol… which is not a platform.
      - deleted 5y ago
        [deleted]
- zug_zug 5y ago
gary17the 5y ago
"But [Apple executives] said they expected any [implication by malicious actors] attacks to be very rare and that in any case a review would then look for other signs of criminal hacking." Oh, all right: Apple, already being in possession of hard evidence of a hideous crime and already being required by law to forward such evidence to proper authorities, will also - pro bono publico! - sacrifice significant amount of time of a significant number of their in-house computer forensics experts, each enjoying a significant billing rate, to relentlessly look for "other signs of criminal hacking", until there is no significant doubt that the accused is, indeed, guilty. We're all safe, then. I don't know whether to laugh or cry.
- zionic 5y ago
  It's going to take NSO spyware putting CP on apple execs personal devices to get them to understand... isn't it? Not that I would ever advocate for such a thing of course.
  - deleted 5y ago
    [deleted]
- tenpies 5y ago
  I just don't understand how a 60 year old gay man doesn't see how insidious this all is, and somehow trusts law enforcement to be such a benevolent actor.
  - adventured 5y ago
    Tim Cook isn't in charge. They're starting the process of complying with, acquiescing to, the multitudes of politicians in the West that are demanding a change in the super structure of online privacy and how it's treated. Anyone here think Yahoo executives actually decided whether the company joined PRISM or not? Those executives also were not in charge. There is a bigger boss in DC, radically more powerful, and most everybody here knows what they're after. They're sick of waiting, they're going to attempt to make another big surveillence move during the relative calm of the Biden Admin (they couldn't do it effectively under Trump, there was too much chaos, the government wasn't functioning very well). What program is actually being put into place right now - that Apple is probably joining up to, as with PRISM - that we won't find out about for many years? It's going to get a lot worse across the board over the coming decade.
deleted 5y ago
[deleted]
arrty88 5y ago
This is apple’s answer to not decrypting / unlocking phones for authorities. They found a way to keep our data private, while still being able to detect criminal activity. Oddly enough, most ISPs already scan for CP on the wire. So not even sure this is a necessary next step.
StreamBright 5y ago
Can't wait to deal with false positives.
kawsper 5y ago
Are there any alternatives to iCloud Photos?
fortran77 5y ago
I don't use Apple devices, but I'm wondering -- are the type of people who use iPhones significantly more likely to be engaged in exchanging child pornography than people on Android? Perhaps there's a real problem here that needs to be addressed (though not in this way that opens the door to all kinds of surveillance)?
- tomschlick 5y ago
  Google (and every other major tech company) already are doing exactly what Apple is. The only difference is that Apple is going to start doing the computing of hashes on the device for photos that are about to be uploaded, vs waiting for them to be uploaded to iCloud. To me this signals that they are going to start allowing E2E encrypted photos on iCloud but they need to compute the hashes on device to comply with the law because they cant hash them once they are encrypted.
ibigb 5y ago
Q1) Is apple responding to government lawmakers: EARN IT seeks to deal with the scourge of online child exploitation by coercing service providers to more aggressively police such content on their platforms. https://www.congress.gov/bill/116th-congress/senate-bill/3398/text https://www.congress.gov/bill/116th-congress/senate-bill/339... Similar laws in UK and others. Maybe this will short circuit the need for a government backdoor to snoop in icloud photos? Q2) Didn't people agree to no illegal KP with the icloud TOS? Doesn't all this do is move the scanning from apple's servers to the distributed ARM processors? Q3) Is that more environmentally friendly or less? I am sure it is cheaper for apple to have the iphone scan than add additional servers, cooling, space, etc. If one doesn't use icloud photos this does not affect them, for now.
uhtred 5y ago
One solution is don't use Apple products?
browningstreet 5y ago
Another thought I've had: I'm not sure hiring tens of thousands of people to look at porn and child porn is really the future solution set that we all want. We'll have another subclass of of our culture, like military vets, who'll have trauma and PTSD as part of their job experience.
- andrewmcwatters 5y ago
  What? No, it’ll be a small amount of contracted resources that become second-class Apple “employees” who aren’t allowed to say they work for Apple. That small group will be responsible for taking on the work, which will be way more than the small number of resources can handle, then Vice will write an article about how traumatic the experience is.
  - browningstreet 5y ago
    My family just watched Hurt Locker last weekend.
pshirshov 5y ago
Well, this morning I got my Pixel 5 delivered. Installed CalyxOS in 5 minutes. Locked bootloader. The experience is not that bad. In-app purchases aren't working, GPay doesn't work either. And the camera is, well, bad. Apart of that everything seems to be smooth and fine. Try it and donate the iPhone price difference to Calyx Institution. You don't even have to give up on your old iPhone and update its OS.
- lifty 5y ago
  I read that you can use the official Pixel camera and block access to the internet, only for that specific app.
  - pshirshov 5y ago
    I'll try it, thanks!
Geee 5y ago
I don't care about the technicalities. The issue is that we would be constantly watched with a government defined black list. They could find all "troublemakers" with a simple query. This gives immense power to goverments, and completely destroys any notion of individual freedom. If you support Apple on this, you support totalitarianism.
robomartin 5y ago
This is one of those things where you can align with the intent --child abuse is a horrible thing-- and yet, at the same time, cringe at the prospect of what doors we might open. I don't use iCloud. I have no need for it. Then again, most people on HN do not fit the profile of the average Apple user. When you are technically capable some of these things don't have the same value they may have for you parent, uncle or grandma. In my case, I had a couple of problems back in the iPhone 3 days and just opted to ignore it completely. Today, my iPhone X isn't using iCloud and all is well. That said, I have seen people do things like take pictures of tax and other documents and message them to others. I can't possibly imagine what people take pictures of and unwittingly keep in their phones and on iCloud. ID, paychecks, that wart in their crotch, anything. The average user has no clue how any of this works. It's magical. And, yes, it's simple. And, yes, it comes with potential consequences. And now, all of it is up for evaluation for potentially criminal activity? By an anonymous a team with no legal accountability to anyone? Without and before being accuse of anything? Wow. What doors are we opening?
nikkinana 5y ago
Finally! They need to check for all kinds of crime. It's illegal. Start with the kids too, they need to know right and wrong from an early age.
freebuju 5y ago
Nothing Apple says about this modern day surveillance tool will make me more accepting of it. If you think this isn't about establishing complete control of your communications, you are a fool. If you think this about protecting the children, you are a bigger fool. I do not want AI making such decisions affecting humans. No matter how good it is. I also don't want John from Apple looking at my profile and assigning me a score on a scale of 1-10 of how "pedo" am likely to actually be. What I actually want is for people to stop thinking that technology will solve every human problem we have. You have to be either naive, conceited or just lazy (avoiding the real work) to actually believe this is possible.
Consultant32452 5y ago
Remember when they let Epstein rape kids for decades? They don't care about CP.
- antocv 5y ago
  They dont want competition.
radicaldreamer 5y ago
Just an FYI for everyone: you can use a local backup system with iOS. Fully encrypted local backups over WiFi (connect your iPhone to your Mac or Windows iTunes, use full backups - encrypted, and enable backups over WiFi). Your phone will backup when charging overnight on the same WiFi network as your designated backup Mac/PC. The backup files are encrypted with a different password chosen when you set it up so it doesn't rely on only keeping your backup computer secure.
- Crosseye_Jack 5y ago
  Which is handy for us geeks who stereotypically would have a higher chance of running a computer 24/7 and so backing up locally can be "just as painless as iCloud" for us. However it doesn't really help the other 99.9% of iOS users. Even if they did, you then have the chance of the user forgetting the password they used to encrypt the data, simply because you only need the password when you a) want to change the password b) use a local backup. Side note - doesn't really have nothing to do with backups :-P: When I took my phone in for a battery swap (Apple did it for free so I didn't botrher replacing it myself) they asked if I had backed up my phone as there was a small tiny chance they would have to wipe the phone. I said I had. When they were booking it into the system the person booking it in questioned me on the backup because his software wasn't showing a backup. The person booking it in was looking for iCloud backups.
- sandstrom 5y ago
  It's great that this exist (and it's good that you're pointing it out!). But it's pretty basic, unfortunately. If Apple would just spend a bit more effort, running iPhone backups to your mac via Wifi regularly would be totally viable. It still is, but not's not very convenient. For example, I cannot exclude certain categories from these backups. I'd like to exclude photos, since they are already on my computer and I don't want needless duplicates of them. Same with e.g. downloaded podcast episodes. Similarly, it seems like I can't backup my contacts, since those are already in iCloud. https://www.reddit.com/r/applehelp/comments/i936ov/how_to_exclude_photos_from_iphone_physical_backup/ https://www.reddit.com/r/applehelp/comments/i936ov/how_to_ex...
  - 5y ago
NotChina 5y ago
An exclusive Concierge Service to outsource our own conscience? No wonder their phones are so much more expensive.
kurizu4444 5y ago
How is this supposed to be helpful? Wouldn't a perpetrator simply turn off iCloud syncing for their photos? Why would they even store them in the photos app in the first place?
- noptd 5y ago
  Exactly. Especially considering it's been announced publicly. Cynical take- as pedos move to other means of storing and sharing CSAM, there will be far fewer photos flagged for review which requires fewer reviewers to be paid by Apple. If they wanted to do this for the greater good as some users claim, wouldn't they have been far more successful in catching criminals if they kept this system secret? Disclaimer: I'm not in support of keeping it secret nor even the system itself, but this is a question worth considering when viewing the situation through the greater good lens.
largehotcoffee 5y ago
What if your iCloud account gets broken into, and a malicious person uploads a bunch of CSAM?
- system2 5y ago
  What if Apple deliberately does this to frame anyone? Or any government officials? This Apple news really changed the way I approach my backups and data.
stiltzkin 5y ago
Maybe this would be an unpopular opinion but for anyone who loves Apple ecosystem you can have an Apple device with minimum private stuff and a secondary non-google phone for private related stuff?.
mensetmanusman 5y ago
How politicians will abuse this: -upload hash of meme you find offensive that the political opposition is using to subvert your authority -receive addresses of ‘offenders’
- smoldesu 5y ago
  The worst part of this is that it's entirely possible (see: inevitable), and there's no way for us to hold Apple accountable. Once Apple starts hashing the rest of their userspace, the government will have access to a low-resolution transactive history containing the signature of every file you ever saved, made or shared. The fact that this data even exists is a sign that it will be abused.
pengaru 5y ago
Just another entry on the list of reasons I'm thrilled to have never given a dime to this company.
tdaltonc 5y ago
Don't like it? Call your Senator. Apple has to comply with the law. "You’re going to find a way to do this or we’re going to do this for you. We’re not going to live in a world where a bunch of child abusers have a safe haven to practice their craft. Period. End of discussion." - Sen. Lindsey Graham The system they recently announced is a step forward from how they currently do it. https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
- matheusmoreira 5y ago
  There's no way to argue with these people. Every time we try, they just double down on the same arguments.
- MonkeyClub 5y ago
  'nakedsecurity' is definitely the wrong hostname for this post -- or the absolutely right one :)
Seattle3503 5y ago
A natural extension of these systems would be to enforce copyright, no? Of course it would be cause press to sue anyone possessing copyrighted content. A more measured response would be to have it disappear from iCloud, with a message that it has been put in the memory hole.
brianzelip 5y ago
What happens when parents record their toddler running around naked singing a song, or dancing, etc.?
- majjam 5y ago
  The hashes of your photos are compared against a database of hashes from known child pornography photos. So your photos would not be flagged.
vineyardmike 5y ago
> And so the threshold allows us to reach that point where we expect a false reporting rate for review of one in 1 trillion accounts per year The actual algorithm isnt accurate to 1/1T - they're claiming the human review process is that acacurate.
- godelski 5y ago
  I think they're being a bit careful with the wording. They are noting a false positive rate, not accuracy. One way to get there would be to only report images with 99.999999% confidence. This would obviously not report a lot of stuff that is actually illicit material too.
literallyaduck 5y ago
Read this as everyone at Apple has easy access all your pictures.
indianpianist 5y ago
"The disclosure came in a series of media briefings in which Apple is seeking to dispel alarm over its announcement last week that it will scan users' phones, tablets and computers for millions of illegal pictures." Yes, this definitely "dispels" my alarm. Thanks, Apple.

461 points by shivbhatt 5y ago

Who needs SWATing when you can send a CP pic (either real or with hash collision as per the thread few days ago) from a virtual overseas number/service and get FBI van to show up as well? What about injecting code into a public website to download same pic into local browser cache without user’s knowledge? The simplicity of the attack vectors here that would trigger the “manual” investigation is just dumbfounding and ripe for abuse/misuse.

bpoyner 5y ago
You could completely wreck somebody's life with this. SWATing will look trivial in comparison.
- scbrg 5y ago
  To be fair, SWATing kills people. Death is generally considered a non trivial and also life wrecking event.
  - headmelted 5y ago
    Honestly I'd rather get shot dead by a SWAT team than implicated for something as atrocious as what this tool is looking for. I imagine many people with a family would feel the same way. It's an abomination that will destroy innocent people. The engineers behind this no doubt think it's fool-proof because they believe they're leagues smarter than any of those pesky naysayers ("hey, we're Apple"). If we've learned anything about Apple this year (as if we needed the reminder) is that their software is nowhere close to as flawless as they seem to think it is.
- headmelted 5y ago
  This. Combined with the unpatched remote-root-via-phone-number disclosed in the Pegasus leak this boils down to a single-click "destroy this person's life" tool.
- donkeyd 5y ago
  If you assume that cops will just arrest people without doing any further research... Then yeah.
  - shakna 5y ago
    > If you assume that cops will just arrest people without doing any further research... Then yeah. Like when they arrested & charged someone for a poor facial recognition match that never had a hope of passing human review? [0] Just glancing at the original photo would have stopped that. Or checking his rock-solid alibi. Neither of those things happened. [0] https://www.wired.com/story/flawed-facial-recognition-system-sent-man-jail/ https://www.wired.com/story/flawed-facial-recognition-system...
  - bpoyner 5y ago
    Oh, the police will get a search warrant, and find exactly what they were told would be on your device. The police aren't in the business of discovering your innocence. It's then up to you and your lawyer to prove you didn't put it on your device. Meanwhile your life will fall apart as you get fired, your wife divorces you, you lose all custody of your kids, etc.
- dzhiurgis 5y ago
  You can already do the same. Send message then call the cops. Just because it's auto-detected now doesn't mean it wasn't possible before.
hughrr 5y ago
This is why I just noped the fuck out of the Apple ecosystem. I won’t support anything which relies on opaque blacklisting to ruin lives. In this example as well on iCloud shared galleries you can upload to other people’s ones you have been invited to. What could possibly go wrong?
- jstx1 5y ago
  Maybe wait to see how it's implemented and how it works first? I really think that the HN crowd is having a giant knee-jerk reaction to all of this.
  - bilekas 5y ago
    Its harder to take back policies like this than it is to object and get them stopped initially. Also people have a habit of 'forgetting' about it later. Until stories of how it is misused are found. And then it's another attack vector we need to be conscious of.
    - ElFitz 5y ago
      And that’s how France still has VAT & revenue taxes. Revenue tax? Have to pay for that expensive WWI war effort, you understand? For all the good it did. Same with the VAT. Have to rebuild after WWII, you understand. We also have an "Exceptional and Temporary Contribution" (CET), recently renamed to "Technical Equilibrium Contribution" (still CET. Smart one, that one). A funny one, for a change? When the Germans invaded in WWII, they changed France's timezone to theirs. After the war, we still called it "the German time". There were talks of going back for a few years… Guess who still has noon at 2pm in the summer, decades later? Change, no matter how ridiculously small or sensical, even when nobody benefits from the status quo (ie the damn timezone) is horrendously difficult. Thus one should always assume that once it’s here, whatever "it" is, it’s here to stay.
      - candiodari 5y ago
        There is still one constant: how the state system cares for victims of child abuse is still the same as in WW2. https://www.kansascity.com/news/special-reports/article238206754.html https://www.kansascity.com/news/special-reports/article23820... You would think money would go into the "backend": caring for kids where the state is responsible for everything BEFORE more money goes into the frontend: finding more kids to throw into the hellhole that is child services. Without the "backend" being in order and working well, raising well-educated, stable kids, the frontend is completely immoral. "Saving" kids from abuse, only to throw them into a slightly different kind of abuse ... if any person did that (e.g. a guy marrying a woman (or I guess vice-versa) with that resulting in that person abusing their new spouse's kids) would be considered a despicable crime. Somehow child services, who do the exact same thing (and they use violence to do it) is not a despicable crime. Somehow just because the state does it, makes such things all a-okay. But frankly this is merely the hole in the justification, all this should merely tell you one thing: any government that doesn't work hard to fix the child services backend does not have children's interests at heart when making these sorts of laws (and mostly they're making budget cuts in the backend, of course). Because fundamentally these laws throw children into the child services system. THAT is the real effect these efforts have on the actual children behind this. THAT is what is meant by "saving kids". And if that system is full of abuse, how is that any better than what paedophiles do? It's not. Which means the state is not attempting to help abused or disadvantaged children. In fact, they're doing the opposite.
jacquesm 5y ago
No code required. <img width=0 height=0> would do the job.
- oleganza 5y ago
  That's fine™. You are just going to redirect blame on the original source, provided you got enough Apple Cash on balance to pay the lawyers and stay out of jail while sorting this out.
- mimsee 5y ago
  I wonder how long it takes until they add a feature to Safari to scan all the <img> <video> <canvas> elements for possibly illegal content. Would be very convenient considering Safari is the only browser engine on iOS.
- paulcole 5y ago
  How is that going to get the image into your iCloud photo storage?
  - jacquesm 5y ago
    It doesn't, but it does get the image into your browser cache and onto your machine.
    - paulcole 5y ago
      So what does that do in the context of this conversation about Apple and iCloud?
  - schoolornot 5y ago
    Content-Disposition: attachment; hit the wrong button, done. It's in your iCloud/Downloads folder.
- madmoose 5y ago
  > <img width=0 height=0> would do the job. No, that's not how the Apple's system works.
mimsee 5y ago
Yes. This reminds me of when typing or receiving certain text would make an iPhone crash. But now having your account deleted makes it a feature. For example Whatsapp automatically downloads media to the camera roll which then get uploaded to iCloud. Of course that can be turned off prior, but this is like what happens with backing up. People want to backup, but don't invest the time in it. That's until it's too late, they lost their data and now want their stuff back.
- laurent92 5y ago
  Backup is a good point: - Apple: “Backup your phone to iCloud, it will be safe there.” - 5 minutes later: “We’ve wiped your account because of a photos of (porn actor here) which is not CP but technically minor at the time she filmed.” - “Also we’ve wiped your iPhone because we couldn’t knowingly let you keep that. Good luck contacting your parents, we’ve deleted your contacts. Good luck! PS: We’ve reported you to the police.” - Also you can’t connect to your iMac now.
  - annamargot 5y ago
    Or photos of your own children. We have a Tumblr set up for family to view pics of the kids. Several photos and videos of our kids when they were under 2 were taken down either temporarily or permanently by their CP algo. These were a pic or video of kids in the bath or without a shirt. In none of them could you see bum or bits. Just a semi naked baby. Algorithms like this get things wrong all the time
    - tpush 5y ago
      This is not the kind of algorithm that Apple is be using. That one only scans for already known CSAM in NCMEC's database.
      - shakna 5y ago
        Which may contain the hashes of their photos, because they've been taken down in the past, which means they probably have been added to certain blacklists that may have been integrated into the blackbox of NCMEC's database.
simondotau 5y ago
None of those attacks would work against the system as described by Apple. The only photos scanned are items in your photo library prior to upload to iCloud. Your browser cache is not scanned. Hash collisions would fail human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM.
- soziawa 5y ago
  > Hash collisions would not pass the human review. About the only consequence I can think of for hash collisions is that the person at Apple who performs the human review step has a slightly nicer day because they were about to look at an image... and then it wasn't CSAM. The whitepapers provided by Apple do not say what the human reviews consists of. They could just look at the hashes to make sure there isn‘t a bug in their system.
  - simondotau 5y ago
    > The whitepapers provided by Apple do not say what the human reviews consists of. At minimum what we know is that each flagged image generates a "safety voucher" which consists of metadata, plus a low-resolution greyscale version of the image. The human review process involves viewing the metadata and thumbnail content enclosed in each safety voucher which cumulatively caused that account to be flagged.
    - foobar33333 5y ago
      A human at Apple likely doesn't get access to anything. I assume it would be part of the police group under strict restrictions checking these.
      - simondotau 5y ago
        The data is not sent to a "police group", it is sent to NCMEC. From Apple's FAQ: Will CSAM detection in iCloud Photos falsely flag innocent people to law enforcement? No. The system is designed to be very accurate, and the likelihood that the system would incorrectly flag any given account is less than one in one trillion per year. In addition, any time an account is flagged by the system, Apple conducts human review before making a report to NCMEC. As a result, system errors or attacks will not result in innocent people being reported to NCMEC.
sschueller 5y ago
My public Wifi captivity portal...
2OEH8eoCRo0 5y ago
Don't worry. I'm sure the police will believe you and help you out. /s What you've described is pretty much the scariest thing I can imagine as far as computer crime goes.
ratww 5y ago
I remember WhatsApp used to save each received image to the iCloud Photo Album. I remember one day going to my album and seeing several memes and pics I had received but never saved. Having 3rd party apps that have access to the photo album being able to do that makes it a bit risky to have iCloud.
- EtienneK 5y ago
  WhatsApp was my first thought as well. Any app that automatically saves photos to iCloud without user interaction is a huge risk.
jbverschoor 5y ago
It's a good method of protecting important documents. Simply add some stamps on top of all documents in case someone steals them
sylens 5y ago
You don't even need to inject code into a public website. There have been no shortage of zero-click exploits for iMessage
modernerd 5y ago
The reported response from Apple offers little reassurance: > The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material. But they said they expected any such attacks to be very rare and that in any case a review would then look for other signs of criminal hacking. What triggers them to look for signs of criminal hacking? Does every manual review process involve such checks? Are they searching device backups for indicators of compromise [IoC]? What if there's no device backup or device image to scan? What if the scan fails to notice IoC? What if the device was compromised after the last backup? What if the device was compromised via physical access? What if the device isn't compromised and the material was pushed maliciously or via drive-by download? It's dangerous to assume that all material on a network-connected device arrived with the consent of the user when it can accept incoming messages from strangers, trick people into downloading files, or be compromised without your knowledge. “That isn't mine” is going to be a tough defence if you can't even take measures to log where content came from. Client-side scanning seems to amplify this issue (which could still happen with cloud storage) because at least cloud storage doesn't generally ship with or integrate deeply with messaging apps, social media, a web browser, QR codes, App Clip Codes[1] etc. The impact might be fairly low right now with the current proposal (images would have to be uploaded to iCloud, so cached browser images don't get scanned as far as we know), but the existence of the non-consensual scan in the first place is worrying, because it means such attacks are only a policy change away. [1] : https://developer.apple.com/design/human-interface-guidelines/app-clips/overview/app-clip-codes/ https://developer.apple.com/design/human-interface-guideline...
- GeekyBear 5y ago
  > The executives acknowledged that a user could be implicated by malicious actors who win control of a device and remotely install known child abuse material. Since Google has been scanning your account for kiddie porn for the past decade, wouldn't this apply equally to Google accounts? >a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-led-to-a-mans-arrest-for-child-porn-was-not-a-privacy-violation/ https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le... All people have to do is email you kiddie porn and Google will have you arrested?
quietbritishjim 5y ago
Something slightly different but very related happened to a senior police officer in the UK. She got sent a WhatsApp message by her sister containing a horrific CP act. It was captioned with a message asking people to circulate it to identify the adult in it, and probably those who sent it around (including the sister) were acting in good faith, but actually it was still illegal to send or even possess it. No doubt the originator of the caption was a deliberate troll. She was found guilty of "possessing an indecent image of a child". [1] She tried to argue that she hadn't noticed the message, but it's not surprising that wasn't believed given that she had immediately replied to her sister saying "please call". She was sentenced to 200 hours community service, and originally sacked from her job but recently reinstated after appealing. [2] It seems that she wasn't immediately in trouble when she received the message ... so long as she had immediately reported her own sister for distributing it, even though it's clear that she hadn't deliberately done anything wrong. (In fact the sister had contacted her to ask what she should do about it. Probably her answer was "don't have already sent it me!") [1] https://www.bbc.co.uk/news/uk-england-london-50476166 https://www.bbc.co.uk/news/uk-england-london-50476166 [2] https://www.bbc.co.uk/news/uk-england-london-57501764 https://www.bbc.co.uk/news/uk-england-london-57501764
- zimpenfish 5y ago
  > a senior police officer in the UK To be fair, this is partially because the laws in the UK are, I think, fairly bonkers strict about CSAM - mere possession, whether you've looked at it or not, whether you downloaded it or not, whether you even know it's there or not, etc., is counted as criminal.
  - zionic 5y ago
    The US is the same.
    - wutbrodo 5y ago
      I believe this is incorrect. > At the same time, because of the First Amendment, child pornography offenses are not "strict liability" crimes like statutory rape: in order to convict a defendant, the government must prove that the defendant knew the material involved the actual abuse of a child https://www.zmolaw.com/child-pornography-faqs# https://www.zmolaw.com/child-pornography-faqs# I've found similar claims on the websites of a few law offices. For some reason, the official DoJ materials are pretty cagey on the topic.
nicce 5y ago
> Who needs SWATing when you can send a CP pic (either real or with hash collision as per the thread few days ago) from a virtual overseas number/service and get FBI van to show up as well? You are talking like collisions are trivial to make. I bet they have had a deep conversations in this area. At first, you would need a real hash to even try (which are hidden). Secondly, to get real material it means that it must be in their database to trigger anything. This tells a lot from sender already, and is worth to tell for police. It is quite easy to prove that someone just send it to you. And one photo is not triggering anything. Besides, sender must know that those photos must go automatically into the cloud to mean anything. > What about injecting code into a public website to download same pic into local browser cache without user’s knowledge? At least US legistlation is precise that user must willingly obtain/download CSAM material, and it must be proved. So this is not harmful for the user in the end. A lot of speculation, but does not really lead for coencequences. Almost every system can be tried to be abused, but does it really mean something, is different story.
- vineyardmike 5y ago
  > At least US legistlation > does not really lead for coencequences Except that a trial, even with an innocent verdict will SUCK and have terrible news stories about you and poison any google search for you with CSAM stories
- themaninthedark 5y ago
  Step 1: Get copies of pictures of targets kid in bath from phone/SNS Step 2: Manipulate pictures so that hash collides with CSAM Step 3: Get pictures back on targets phone so they get scanned. I don't have the skills or understanding of how the hashes are created but would this be possible? >At first, you would need a real hash to even try (which are hidden). How are the hashes hidden? It looks like they are shared: https://www.thorn.org/reporting-child-sexual-abuse-content-shared-hash/ https://www.thorn.org/reporting-child-sexual-abuse-content-s...
  - nicce 5y ago
    > How are the hashes hidden? It looks like they are shared: https://www.thorn.org/reporting-child-sexual-abuse-content-s https://www.thorn.org/reporting-child-sexual-abuse-content-s.. These hashes are not generated by Apple and are not valid. (Must be generated by their new system) They are probably very strictly guarded. They will be stored on every iOS from 15 version, somehow securely. This must limit the support of older iPhones.
robertoandred 5y ago
Just because you assume attack vectors are simple doesn't mean they are. First of all, why would Apple forward a report about something that isn't CSAM to the NCMEC?

988747 5y ago

And what if I have iCloud storage turned off? Will this local scanning still be applied?

bkishan 5y ago
Nope, only for photos in iCloud and destined for upload to iCloud.
- roody15 5y ago
  yeup wink wink… the scanner will still be there … but Apple promises not to use it for anything else and only with photos going to upload. Wheww that makes me feel better
- echelon 5y ago
  Apple isn't trustworthy. I bet this stance changes in under five years.
  - madmoose 5y ago
    > Apple isn't trustworthy. Why not? They've told you in detail what they're going to do.
    - xvector 5y ago
      > Why not backtracking on promises after buying from them
    - SheinhardtWigCo 5y ago
      Lando Calrissian: That wasn't part of the deal! You said the wookiee and Leia would stay under my protection! Darth Vader: I am altering the deal. Pray I don't alter it any further. [departs in an elevator]
- sneak 5y ago
  There's no point in clientside scanning, then. The photos in iCloud are already not e2e and are completely readable by Apple (and are regularly turned over to the USG without probable cause or a search warrant).

verytrivial 5y ago

I've read this whole situation as a signal to China and other authoritarian regimes that Apple has finally seen diminishing returns from the "Apple is more secure" angle and is now looking elsewhere for growth. It's just business.

SquishyPanda23 5y ago
I think there is more going on here. I think what we're seeing is Apple betting on using cryptography as part of the product design phase. Apple devices already do weird things like wake up to announce their physical location so that users can find their devices. The thought of a powered down or suspended laptop waking up to announce its location isn't something I particularly want, but Apple users seem to like it. Anyone who has spent any time on spaces that are strongly encrypted and focused on privacy know how quickly they become havens for the sort of material that Apple doesn't want associated with its brand. How many "Apple protects child predator" news stories do you think Apple can withstand while still remaining a luxury brand? Apples goal here is to have the reputation for end-to-end encryption and privacy while simultaneously not being seen as a phone for child predators. They don't have a lot of options if they want to thread that needle. I've thought about this space quite a bit, and all options suck. Client side scanning is really the only choice with reasonable tradeoffs. The other option is scanning encrypted photos on cloud using secure enclaves to do the scanning. My guess is that when the tech makes that possible Apple will move in that direction. I agree that this isn't the best for privacy nuts like me. But the iPhone isn't a blackphone, it's a luxury handbag. The phone isn't for privacy nerds, the privacy is there to make other mobile OS's look cheap and tacky.
- zionic 5y ago
  This is needless Apple apologetics. They deserve to be raked over the coals for this, there's no world where their current design is a "good" or "right" one. Child abuse is a serious problem, but building a surveillance panopticon is not an acceptable solution to it. Better investment in education, health care, and reporting hotlines are the way forward to stop this issue at its source.

Odenwaelder 5y ago

Maybe someone can comment on this: Does Google scan the cloud fotos of its users for CP? Have we seen an uptick of false positives/SWATings since they do that? Apple is - rightfully and understandably IMO - criticized for their plans, but does anyone know how Google handles this?

soziawa 5y ago
There is some previous discussion about this regarding Facebook here: https://news.ycombinator.com/item?id=28119372 https://news.ycombinator.com/item?id=28119372 The false positive rate does not look great.
sneak 5y ago
Google and FB both scan storage for several different types of contraband, and also have triggers and thresholds for things that use too much bandwidth (eg pirated software download links that are shared widely et c).
bengale 5y ago
Yes they do. The reason apple has done this is because they lagged behind other providers considerably in detecting this sort of content. Facebook for example are reporting millions per year compared to a few hundred for Apple. Instead of scanning you whole library they came up with a way to do it on device, which is the main difference between other services. If you don't enable iCloud photo storage the system can't work at all. Very interesting stuff in their technical explanation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- zionic 5y ago
  -Apple "lagged behind" because it built private and secure services it could not monitor by design. This is a feature, not a bug. -Facebook's reporting overwhelmingly flags burner accounts signed up via tor etc, only absolute idiots would post actual CP on their real name account on facebook. -Apples solution is highly invasive and dangerous, and your statement about "only running with iCloud upload" is false. It took less than a week for Apple to announce that they will open these APIs to 3rd party apps.

new299 5y ago

Apple have already been doing this for sometime: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... This is about the new PSI system: https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ In which photos are scanned on the users device. This appears to be a report of a new press conference after the initial announcement? Does anyone have a transcript of this press conference?

techbio 5y ago

Apple is smart enough to determine what's in an photograph, but if I paste text into a sentence in iOS I still need to manually add spaces and format punctuation. I will never forget the time iTunes deleted my music library, or it's inability to deduplicate identical songs. Power.

mabedan 5y ago
do this day, if you switch the country of your apple account, you will lose your entire music library.
- sneak 5y ago
  Cloud music libraries aren't yours to begin with, even when working right. Torrent the actual files.
DavideNL 5y ago
Even worse, iTunes deleted my Music and all my precious playlists, and because it was stored in the cloud it wasn’t included in my Macbook backups… I was so pissed of, i’ve avoided iTunes & Apple Music ever since.
insickness 5y ago
I hate hate hate iTunes. My biggest gripe with iPhone has always been that I don't have direct access to my OS files. I have to spend an inordinate amount of time 'syncing' my device instead of just copying a few tracks to it.
- robomartin 5y ago
  > I hate hate hate iTunes Yup. Same here. This is why I don't have my large collection of CD's in my phone and just use Pandora. Going back to iPhone 3 days, iTunes did not allow me to import my CD's into my device and play them as entire CD's. If I want to listen to Mozart's "Eine kleine Nachtmusik" or Pink Floyd's "The Wall", it's a nightmare. iTunes is song-based, not album based. Well, the above, and many others, are works you pretty much listen to in order as recorded. In some cases (The Wall, Brandenburg Concertos, etc.) the works span multiple CD's. I stopped using iTunes and storing music on my iPhone because of this. I don't enjoy music the way Apple seems to think you should. I have no clue if they fixed this since iPhone 3 days. I would not be surprised if they have not. In sharp contrast to this, I have not problem playing single or multi-CD works as intended using Windows Media Player on my desktop, where I have my entire CD collection stored. This, for me, is the single reason I would instantly jump into a Windows phone if Microsoft got their heads out of their asses, committed to doing a good job and integrated a phone experience with the desktop. They would have to regain my trust, but as a life-long user of both Apple and MS desktop products, I would absolutely welcome a better phone experience than Apple has delivered over the years. I really want to abandon iPhone and go to a good Windows phone, but MS does not seem interested in creating that opportunity. Oh, yes, and to address iCloud, back in the early days it managed to delete not only whatever I had on iTunes (which I own on CD's so I don't care) but all of my contacts. Thankfully I had my contacts stored in my prior phone (I think it was a Blackberry). After disconnecting from iCloud I entered them manually and never again enabled iCloud all the way up to my current iPhone X.

deleted 5y ago

[deleted]

endisneigh 5y ago

Which cloud provider doesn’t do this and would be unable to do this at any point in the future?

TroisM 5y ago
Any of them as long as you encrypt before you upload.
lazzurs 5y ago
Proton. https://protonmail.com/blog/protondrive-security/ https://protonmail.com/blog/protondrive-security/
- foobiekr 5y ago
  The first stage of their receive pipeline has access to the plaintext. Ditto the send pipeline.

salmo 5y ago

Wow. This could be messed up for attorneys, DCS, social workers, etc. They allude more to child pornography, but I hope it doesn't extend to physical abuse. Those photos are usually taken on phones by spouses, doctors, schools, etc. to be passed to the above on their phone for evidence for a DNN or similar case. Glad my kids have aged out of baby bath photos. And those poor people who I know are going to have to provide an auditing safeguard. I hope they take care of their mental health.

judge2020 5y ago
In every thread about this, someone makes this same false assumption: no, Apple is not scanning for naked children or children in pain. It's generating a hash to be compared against hashes of NCMEC-verified CSAM pictures (and while some HN commenters claim the DB contains non-CSAM, that has not been verified nor ever reported on by a news publication) and it does indeed only scan photos destined for iCloud Photos (which I theorize is the only part keeping this system legal[0]). 0: https://news.ycombinator.com/item?id=28112982 https://news.ycombinator.com/item?id=28112982
- zionic 5y ago
  >that has not been verified nor ever reported on by a news publication) How could it be? The list is literally property of the secret police, you can't know what's on it. No one can audit it except the police themselves.
  - judge2020 5y ago
    Auditing it would mean looking at CSAM, which is illegal. If it was widespread, I would expect at least one of the manual reviewers that are able to legally view the CSAM would contact major publications (under the promise of staying anonymous) and whistleblow on this issue. : To be clear, I'd expect whistleblowing if these manual reviewers were tasked to 'accept' CSAM submissions that aren't CSAM.
    - zionic 5y ago

beshrkayali 5y ago

I stopped using stock Android and went back to iPhones because I thought Apple cares more about privacy than Google does. Not exactly correct in all cases I know (ie. they both suck in terms of privacy), but it seemed like Apple users are buying in, so it might work. Now I think my next mobile OS is going to be GrapheneOS. Like others have mentioned, this is as big of a warning as anyone's going to get to get out of that locked-in ecosystem. On that note, the outrage is kind of useless if you don't skip buying the next iPhone. You should fully own what you fully pay for.

Maxburn 5y ago
Hail to Graphene being the official No Agenda Phone https://noagendaphone.com https://noagendaphone.com
- tbihl 5y ago
  Why disable 5G? I was following everything until that point, but I'm not sure the reason for that choice.
  - dimitrios1 5y ago
    I am curious as well. My guesses are a. 5G allows for metadata collection points b. 5G chips are much more expensive because of patents
    - Maxburn 5y ago
      Me too. A; maybe, I'm not sure what extra that does above LTE. B; I guess you could get a different phone that doesn't have it, for a little while longer.
  - Maxburn 5y ago
    I'm not sure myself, I just know they push Graphene pretty heavily on the No Agenda show.
  - da_big_ghey 5y ago

hkai 5y ago

So... Any predictions for how long till this is used to target opposition?

hkai 5y ago

I am totally making an npm package that secretly generates a CP image during the postinstall script. Developers created this and developers must suffer from this.

zionic 5y ago
I know this is a joke post, but I think in all seriousness it's going to take a crisis like this to get these laws/society changed. "Thousands of developers swept up in CP ring!" that later turns out to be malware planting CP would go a long way towards fixing this issue. I really am surprised nobody has made a worm that's sole function is to hit every FBI honeypot in existence and archive it to hidden folders just to prove a point.

EGreg 5y ago

I find a marked difference on HN about the attitude towards end-to-end encryption, and anonymous transactions in cryptocurrencies. The former can enable the latter. And so much more. Organizing sex trafficking, terrorism and so forth. Nevermind the copyright protection stuff. The latter can enable tax evasion, money laundering and financing unsavory activities. States don’t want people to be able to do that. Yet many on HN applaud attempts to doxx everyone and every transaction in crypto, calling it a scam/for criminals, while at the same time decry any attempts to lessen encryption, however subtle or careful, of personal files and communication. What is a consistent position on both these topics, given that there are dangers on both sides of the argument? I tried to present the core issue here: https://news.ycombinator.com/item?id=28117289 https://news.ycombinator.com/item?id=28117289

CTDOCodebases 5y ago

So Apple wants to use my electricity/computation instead of doing it on their own servers. Disregarding the privacy issues that’s pretty scummy IMHO.

GeekyBear 5y ago
Google and the rest do the same thing, but on their own servers. You want Google, for example, to hold false positive data on their servers forever where it can be subpoenaed and misused? >Innocent man, 23, sues Arizona police for $1.5million after being arrested for murder and jailed for six days when Google's GPS tracker wrongly placed him at the scene of the 2018 crime https://www.dailymail.co.uk/news/article-7897319/Police-arrested-innocent-man-murder-using-Google-location-data.html https://www.dailymail.co.uk/news/article-7897319/Police-arre...
- CTDOCodebases 5y ago
  What’s not to say the logs of the scans being performed on my device will be uploaded and stored off my device forever anyway? The point I was trying to make was privacy reasons aside their motivation of doing it on the users device is scummy. Why don’t they mine Bitcoin on my iPhone while they are at it?
  - GeekyBear 5y ago
    If there is a false positive, I don't want that fact to ever leave my phone, instead of residing on Google's servers forever, where it can be subpoenaed and misused. Apple's approach here is far superior from a privacy standpoint. >1. Only if you're uploading files are the files matched. 2. Only if the matches are very close are they considered matches. 3. Only if you have multiple very close matches is Apple able to decrypt the low-res versions of the images themselves. 4. Only if a human reviewer discovers any of the decrypted low-res images to be illegal content is any of your information shared with anyone else. https://news.ycombinator.com/item?id=28120598 https://news.ycombinator.com/item?id=28120598
    - 5y ago

ddtaylor 5y ago

I'm confused. I was under the assumption that they were only going to do client side detection, is this article claiming they will be running the scans in the cloud as well?

bengale 5y ago
No, they are putting this in place to avoid doing this in the cloud against your whole library and having access to all of the information that generates. https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

squarefoot 5y ago

> "But they said they expected any such attacks to be very rare ..." Well, ransomware has been rare almost forever, then suddenly became the norm. > "and that in any case a review would then look for other signs of criminal hacking." Good luck finding a malicious app that downloads child porn from an encrypted remote server, plants it in the target device, sends "by mistake" an example to social media using the owner credentials, then deletes itself. This is crazy. Child porn traffickers will find a way to circumvent this while it would offer governments just another weapon against people they don't like. Also they completely ignore that we're talking about child porn; if someone is wrongly linked with the subject for just one second by the media, no matter how many times the news is being rectified afterwards, his life may be ruined forever. It's not like being accused of avoiding taxes or theft; any mental association with things like child porn or rape is not going away easily. Any technology that could be (ab)used to plant evidence in such cases would be the ultimate weapon to destroy individuals without actually killing them. Better not to have it than to risk that it ends in the wrong hands.

onepunchedman 5y ago

Can someone please explain to me how this comparison would work? It seems so trivial to alter any image containing CP slightly such that its hash doesn't compare anymore?

sneak 5y ago
Perceptual hashes are not related to byte-level hashes.
- onepunchedman 5y ago
  Yeh I must have ignored the "perceptual" part when I read it over
dilap 5y ago
Confusingly, it's a completely different use of the word "hash" See here: https://en.wikipedia.org/wiki/Perceptual_hashing https://en.wikipedia.org/wiki/Perceptual_hashing The goal of a perceptual hash is to generate a number that will be the same for all "similar" looking images. Think like what Shazam does, but in the visual domain.
- onepunchedman 5y ago
  Thanks for the response, I was super confused by this part!
bengale 5y ago
https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- onepunchedman 5y ago
  Thanks for this! :)

3pt14159 5y ago

I had a highly upvoted comment about this whole thing a couple of days ago: https://news.ycombinator.com/item?id=28069528 https://news.ycombinator.com/item?id=28069528 I have come to one more nuance about the viewpoint. If people are spreading CP by signing into the same Apple account from multiple devices and using iCloud to automatically share the photos, I think that's a different situation than a single person signing into one computer and one phone that are mostly used on the same networks together. Not that I've really changed my view that I wrote before, just there is a bit of grey here.

ericwooley 5y ago
A) that would be clever, and if the bad guys are that clever, they will easily find another way to share their stuff. B) apple could simply make a policy that your Icloud will trigger a scan / review after x devices are logged into it within a certain amount of time, in order to make sure you aren't using Icloud as a distribution platform for something. This whole scanning locally on device wouldn't be necessary.

tomasreimers 5y ago

If you haven't seen it already, Alex Stamos (former head of Security for FB) has a really incredible Twitter thread on this: https://twitter.com/alexstamos/status/1424054568275439617 https://twitter.com/alexstamos/status/1424054568275439617

jiggawatts 5y ago
"Something went wrong. Try reloading."
zionic 5y ago
Wow, he's actually suggesting Apple should build a "trust and safety" team for reporting E2E Encrypted messages containing "hate speech" to directly? Sometimes I forget how messed up Twitter is.
- fsociety 5y ago
  Yes because it’s a mechanism to prevent actual abuse from occurring on their platform and to report offenders directly. When one of the participants of an E2EE conversation reports the convo then the messages would be sent up in a way for Trust and Safety to read the messages and report to the authorities. What is messed up about that? The method of reporting is in the hands of the user, not an ML algorithm. The ML algorithm would prompt the kid to stop and think about what is happening, before actual abuse occurs.. I assure you Stamos is speaking from a place of experience, in having to prevent these kind of things.
  - zionic 5y ago
    My $1000 hardware is my property, not “Apple’s platform”. Also, messaging is a protocol… which is not a platform.
    - deleted 5y ago
      [deleted]
zug_zug 5y ago

gary17the 5y ago

"But [Apple executives] said they expected any [implication by malicious actors] attacks to be very rare and that in any case a review would then look for other signs of criminal hacking." Oh, all right: Apple, already being in possession of hard evidence of a hideous crime and already being required by law to forward such evidence to proper authorities, will also - pro bono publico! - sacrifice significant amount of time of a significant number of their in-house computer forensics experts, each enjoying a significant billing rate, to relentlessly look for "other signs of criminal hacking", until there is no significant doubt that the accused is, indeed, guilty. We're all safe, then. I don't know whether to laugh or cry.

zionic 5y ago
It's going to take NSO spyware putting CP on apple execs personal devices to get them to understand... isn't it? Not that I would ever advocate for such a thing of course.
- deleted 5y ago
  [deleted]
tenpies 5y ago
I just don't understand how a 60 year old gay man doesn't see how insidious this all is, and somehow trusts law enforcement to be such a benevolent actor.
- adventured 5y ago
  Tim Cook isn't in charge. They're starting the process of complying with, acquiescing to, the multitudes of politicians in the West that are demanding a change in the super structure of online privacy and how it's treated. Anyone here think Yahoo executives actually decided whether the company joined PRISM or not? Those executives also were not in charge. There is a bigger boss in DC, radically more powerful, and most everybody here knows what they're after. They're sick of waiting, they're going to attempt to make another big surveillence move during the relative calm of the Biden Admin (they couldn't do it effectively under Trump, there was too much chaos, the government wasn't functioning very well). What program is actually being put into place right now - that Apple is probably joining up to, as with PRISM - that we won't find out about for many years? It's going to get a lot worse across the board over the coming decade.

deleted 5y ago

[deleted]

arrty88 5y ago

This is apple’s answer to not decrypting / unlocking phones for authorities. They found a way to keep our data private, while still being able to detect criminal activity. Oddly enough, most ISPs already scan for CP on the wire. So not even sure this is a necessary next step.

StreamBright 5y ago

Can't wait to deal with false positives.

kawsper 5y ago

Are there any alternatives to iCloud Photos?

fortran77 5y ago

I don't use Apple devices, but I'm wondering -- are the type of people who use iPhones significantly more likely to be engaged in exchanging child pornography than people on Android? Perhaps there's a real problem here that needs to be addressed (though not in this way that opens the door to all kinds of surveillance)?

tomschlick 5y ago
Google (and every other major tech company) already are doing exactly what Apple is. The only difference is that Apple is going to start doing the computing of hashes on the device for photos that are about to be uploaded, vs waiting for them to be uploaded to iCloud. To me this signals that they are going to start allowing E2E encrypted photos on iCloud but they need to compute the hashes on device to comply with the law because they cant hash them once they are encrypted.

ibigb 5y ago

Q1) Is apple responding to government lawmakers: EARN IT seeks to deal with the scourge of online child exploitation by coercing service providers to more aggressively police such content on their platforms. https://www.congress.gov/bill/116th-congress/senate-bill/3398/text https://www.congress.gov/bill/116th-congress/senate-bill/339... Similar laws in UK and others. Maybe this will short circuit the need for a government backdoor to snoop in icloud photos? Q2) Didn't people agree to no illegal KP with the icloud TOS? Doesn't all this do is move the scanning from apple's servers to the distributed ARM processors? Q3) Is that more environmentally friendly or less? I am sure it is cheaper for apple to have the iphone scan than add additional servers, cooling, space, etc. If one doesn't use icloud photos this does not affect them, for now.

uhtred 5y ago

One solution is don't use Apple products?

browningstreet 5y ago

Another thought I've had: I'm not sure hiring tens of thousands of people to look at porn and child porn is really the future solution set that we all want. We'll have another subclass of of our culture, like military vets, who'll have trauma and PTSD as part of their job experience.

andrewmcwatters 5y ago
What? No, it’ll be a small amount of contracted resources that become second-class Apple “employees” who aren’t allowed to say they work for Apple. That small group will be responsible for taking on the work, which will be way more than the small number of resources can handle, then Vice will write an article about how traumatic the experience is.
- browningstreet 5y ago
  My family just watched Hurt Locker last weekend.

pshirshov 5y ago

Well, this morning I got my Pixel 5 delivered. Installed CalyxOS in 5 minutes. Locked bootloader. The experience is not that bad. In-app purchases aren't working, GPay doesn't work either. And the camera is, well, bad. Apart of that everything seems to be smooth and fine. Try it and donate the iPhone price difference to Calyx Institution. You don't even have to give up on your old iPhone and update its OS.

lifty 5y ago
I read that you can use the official Pixel camera and block access to the internet, only for that specific app.
- pshirshov 5y ago
  I'll try it, thanks!

Geee 5y ago

I don't care about the technicalities. The issue is that we would be constantly watched with a government defined black list. They could find all "troublemakers" with a simple query. This gives immense power to goverments, and completely destroys any notion of individual freedom. If you support Apple on this, you support totalitarianism.

robomartin 5y ago

This is one of those things where you can align with the intent --child abuse is a horrible thing-- and yet, at the same time, cringe at the prospect of what doors we might open. I don't use iCloud. I have no need for it. Then again, most people on HN do not fit the profile of the average Apple user. When you are technically capable some of these things don't have the same value they may have for you parent, uncle or grandma. In my case, I had a couple of problems back in the iPhone 3 days and just opted to ignore it completely. Today, my iPhone X isn't using iCloud and all is well. That said, I have seen people do things like take pictures of tax and other documents and message them to others. I can't possibly imagine what people take pictures of and unwittingly keep in their phones and on iCloud. ID, paychecks, that wart in their crotch, anything. The average user has no clue how any of this works. It's magical. And, yes, it's simple. And, yes, it comes with potential consequences. And now, all of it is up for evaluation for potentially criminal activity? By an anonymous a team with no legal accountability to anyone? Without and before being accuse of anything? Wow. What doors are we opening?

nikkinana 5y ago

Finally! They need to check for all kinds of crime. It's illegal. Start with the kids too, they need to know right and wrong from an early age.

freebuju 5y ago

Nothing Apple says about this modern day surveillance tool will make me more accepting of it. If you think this isn't about establishing complete control of your communications, you are a fool. If you think this about protecting the children, you are a bigger fool. I do not want AI making such decisions affecting humans. No matter how good it is. I also don't want John from Apple looking at my profile and assigning me a score on a scale of 1-10 of how "pedo" am likely to actually be. What I actually want is for people to stop thinking that technology will solve every human problem we have. You have to be either naive, conceited or just lazy (avoiding the real work) to actually believe this is possible.

Consultant32452 5y ago

Remember when they let Epstein rape kids for decades? They don't care about CP.

antocv 5y ago
They dont want competition.

radicaldreamer 5y ago

Just an FYI for everyone: you can use a local backup system with iOS. Fully encrypted local backups over WiFi (connect your iPhone to your Mac or Windows iTunes, use full backups - encrypted, and enable backups over WiFi). Your phone will backup when charging overnight on the same WiFi network as your designated backup Mac/PC. The backup files are encrypted with a different password chosen when you set it up so it doesn't rely on only keeping your backup computer secure.

Crosseye_Jack 5y ago
Which is handy for us geeks who stereotypically would have a higher chance of running a computer 24/7 and so backing up locally can be "just as painless as iCloud" for us. However it doesn't really help the other 99.9% of iOS users. Even if they did, you then have the chance of the user forgetting the password they used to encrypt the data, simply because you only need the password when you a) want to change the password b) use a local backup. Side note - doesn't really have nothing to do with backups :-P: When I took my phone in for a battery swap (Apple did it for free so I didn't botrher replacing it myself) they asked if I had backed up my phone as there was a small tiny chance they would have to wipe the phone. I said I had. When they were booking it into the system the person booking it in questioned me on the backup because his software wasn't showing a backup. The person booking it in was looking for iCloud backups.
sandstrom 5y ago
It's great that this exist (and it's good that you're pointing it out!). But it's pretty basic, unfortunately. If Apple would just spend a bit more effort, running iPhone backups to your mac via Wifi regularly would be totally viable. It still is, but not's not very convenient. For example, I cannot exclude certain categories from these backups. I'd like to exclude photos, since they are already on my computer and I don't want needless duplicates of them. Same with e.g. downloaded podcast episodes. Similarly, it seems like I can't backup my contacts, since those are already in iCloud. https://www.reddit.com/r/applehelp/comments/i936ov/how_to_exclude_photos_from_iphone_physical_backup/ https://www.reddit.com/r/applehelp/comments/i936ov/how_to_ex...
- 5y ago

NotChina 5y ago

An exclusive Concierge Service to outsource our own conscience? No wonder their phones are so much more expensive.

kurizu4444 5y ago

How is this supposed to be helpful? Wouldn't a perpetrator simply turn off iCloud syncing for their photos? Why would they even store them in the photos app in the first place?

noptd 5y ago
Exactly. Especially considering it's been announced publicly. Cynical take- as pedos move to other means of storing and sharing CSAM, there will be far fewer photos flagged for review which requires fewer reviewers to be paid by Apple. If they wanted to do this for the greater good as some users claim, wouldn't they have been far more successful in catching criminals if they kept this system secret? Disclaimer: I'm not in support of keeping it secret nor even the system itself, but this is a question worth considering when viewing the situation through the greater good lens.

largehotcoffee 5y ago

What if your iCloud account gets broken into, and a malicious person uploads a bunch of CSAM?

system2 5y ago
What if Apple deliberately does this to frame anyone? Or any government officials? This Apple news really changed the way I approach my backups and data.

stiltzkin 5y ago

Maybe this would be an unpopular opinion but for anyone who loves Apple ecosystem you can have an Apple device with minimum private stuff and a secondary non-google phone for private related stuff?.

mensetmanusman 5y ago

How politicians will abuse this: -upload hash of meme you find offensive that the political opposition is using to subvert your authority -receive addresses of ‘offenders’

smoldesu 5y ago
The worst part of this is that it's entirely possible (see: inevitable), and there's no way for us to hold Apple accountable. Once Apple starts hashing the rest of their userspace, the government will have access to a low-resolution transactive history containing the signature of every file you ever saved, made or shared. The fact that this data even exists is a sign that it will be abused.

pengaru 5y ago

Just another entry on the list of reasons I'm thrilled to have never given a dime to this company.

tdaltonc 5y ago

Don't like it? Call your Senator. Apple has to comply with the law. "You’re going to find a way to do this or we’re going to do this for you. We’re not going to live in a world where a bunch of child abusers have a safe haven to practice their craft. Period. End of discussion." - Sen. Lindsey Graham The system they recently announced is a step forward from how they currently do it. https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...

matheusmoreira 5y ago
There's no way to argue with these people. Every time we try, they just double down on the same arguments.
MonkeyClub 5y ago
'nakedsecurity' is definitely the wrong hostname for this post -- or the absolutely right one :)

Seattle3503 5y ago

A natural extension of these systems would be to enforce copyright, no? Of course it would be cause press to sue anyone possessing copyrighted content. A more measured response would be to have it disappear from iCloud, with a message that it has been put in the memory hole.

brianzelip 5y ago

What happens when parents record their toddler running around naked singing a song, or dancing, etc.?

majjam 5y ago
The hashes of your photos are compared against a database of hashes from known child pornography photos. So your photos would not be flagged.

vineyardmike 5y ago

> And so the threshold allows us to reach that point where we expect a false reporting rate for review of one in 1 trillion accounts per year The actual algorithm isnt accurate to 1/1T - they're claiming the human review process is that acacurate.

godelski 5y ago
I think they're being a bit careful with the wording. They are noting a false positive rate, not accuracy. One way to get there would be to only report images with 99.999999% confidence. This would obviously not report a lot of stuff that is actually illicit material too.

literallyaduck 5y ago

Read this as everyone at Apple has easy access all your pictures.

indianpianist 5y ago

"The disclosure came in a series of media briefings in which Apple is seeking to dispel alarm over its announcement last week that it will scan users' phones, tablets and computers for millions of illegal pictures." Yes, this definitely "dispels" my alarm. Thanks, Apple.