22 ms·

Apple clarifies why it abandoned plan to detect CSAM in iCloud photos

224 points by Anonboxis 3y ago

_boffin_ 3y ago
So… they’re just doing on device scanning instead of icloud and just calling it a different name?
- meepmorp 3y ago
  Not really? It looks like the nudity detection features are all on device, aren't CSAM specific, and seem to be mostly geared towards blocking stuff like unsolicited dick pics. The earlier design was a hybrid model that scanned for CSAM on device, then flagged files were reviewed on upload.
- olliej 3y ago
  No, the terrible misfeature that this group wants is “government provides a bunch of opaque hashes that are ‘CSAM’, all images are compared with those hashes, and if the hashes match then the user details are given to police” Note that by design the hashes cannot be audited (though in the legitimate case I don’t imagine doing so would be pleasant), so there’s nothing stopping a malicious party inserting hashes of anything they want - and then the news report will be “person x bought in for questioning after CSAM detector flagged them”. That’s before countries just pass explicit laws saying that the filter must includE LGBT content (in the US several states consider books with lgbt characters to be sexual content, so a lgbt teenager would be de facto CSAM), in the UK the IPA is used to catch people not collecting dog poop so trusting them not to expand scope is laughable, in Iran a picture of a woman without a hijab would obviously be reportable, etc What Apple has done is add the ability to filter content (eg block dick picks) and for child accounts to place extra steps (incl providing contact numbers I think?) if a child attempts to send pics with nudity, etc
  - Dig1t 3y ago
    >in the UK the IPA is used to catch people not collecting dog poop What does this mean? What is IPA? I tried Googling for it but I’m not finding much. I would love to learn more about that
    - olliej 3y ago
      The investigatory powers act. It was passed to stop terrorism, because previously they found that having multiple people (friends and family etc) report that someone was planning a terrorist attack failed to stop a terrorist attack.
  - robertoandred 3y ago
    How exactly would you be able to "filter" LGBT content? I don't think you understand how this system would've worked.
    - olliej 3y ago
      Hypothetically you have hashes for two people of gender X (lets be honest, based on popularity of different types of porn two men). This is not meaningfully different from opaque hash of "CSAM". But you're missing the point: Step 1. generate some opaque hash of the "semantics" of an image Step 2. compare those hashes to some list of hashes of "CSAM", which again fundamentally cannot be audited Step 3. report any hits to law enforcement Step 4. person X is being investigated due to reported violations of laws against child abuse. Basically: how do you design a system in which the state provides "semantic" hashes of "CSAM" that cannot be trivially abused by inclusion of non-CSAM as "CSAM", or by laws mandating inclusion of things that are objectively not-CSAM. Hypothetically: hashes that match christian crosses, star of David, muslim star and/or crescent, etc. Or in the US DNC, RNC, pride, etc flags. Recall that definitionally no one can audit the hashes that would trigger notifying law encforcement.
      - robertoandred 3y ago
        Except this system wouldn't have looked at "semantics". You can't simply match a hash of a cross or star or flag, you have to match a specific photograph. Which photograph do you use?
- sheepscreek 3y ago
  Yes - and there’s a huge difference between the two. In a word, decentralization. By detecting unsafe material on-device / while it is being created, they can prevent it from being shared. And because this happens on individual devices, Apple doesn’t need to know what’s on people’s iCloud. So they can offer end-to-end encryption, where even the data on their servers is encrypted. Only your devices can “see” it (it’s a black box for Apple servers, gibberish - without the correct decryption key).
formerly_proven 3y ago
The who is often interesting with these stories. > a new child safety group known as Heat Initiative Doesn't even have a website or any kind of social media presence; it literally doesn't appear to exist apart from the reporting on Apple's response to them, which is entirely based on Apple sharing their response with media, not the group interacting with media. > Sarah Gardner on the other hand previously appeared as the VP of External Affairs (i.e. Marketing) of Thorn (formerly DNA Foundation): https://www.thorn.org/blog/searching-for-a-child-in-a-private-world-thorn-vp-of-external-affairs-speaks-at-tedxwarwick/ https://www.thorn.org/blog/searching-for-a-child-in-a-privat... So despite looking a bit fishy at first, this doesn't seem to come from a christofascist group.
- figlett 3y ago
  > So despite looking a bit fishy at first, this doesn't seem to come from a christofascist group. Why would you assume this in the first place?
  - bsenftner 3y ago
    They use hysteria to generate power in society.
  - krapp 3y ago
    The main impetus behind "child safety" advocacy nowadays seem to be by cells of extremist right-wing Christian / QAnon types who believe in conspiracy theories like Pizzagate and the "gay groomer" panic. It's a reasonable assumption to make about any such group mentioned in the media that doesn't have an established history at least prior to 2016.
    - figlett 3y ago
      It sounds like an entirely unreasonable assumption to me. Advocating for child safety is something that transcends political differences, and generally unifies people across the political spectrum. I mean, there aren't many people who want paedophiles to be able to amass huge collections of child abuse imagery from other paedophiles online. And pretty much every parent wants their child to be kept safe from predators both online and offline.
gnfargbl 3y ago
> “Scanning every user’s privately stored iCloud data would create new threat vectors for data thieves to find and exploit," Neuenschwander wrote. "It would also inject the potential for a slippery slope of unintended consequences. Scanning for one type of content, for instance, opens the door for bulk surveillance and could create a desire to search other encrypted messaging systems across content types.” Both of these arguments are absolutely, unambiguously, correct. The other side of the coin is that criminals are using E2EE communication systems to share sexual abuse material in ways and at rates which they were not previously able to. This is, I argue, a bad thing. Is is bad for the individuals who are re-victimised on every share. It is also bad for the fabric of society at large, in the sense that if we don't clearly take a stand against abhorrent behaviour then we are in some sense condoning it. Does the tech industry have any alternate solutions that could functionally mitigate this abuse? Does the industry feel that it has any responsibility at all to do so? Or do we all just shout "yay, individual freedom wins again!" and forget about the actual problem that this (misguided) initiative was originally aimed at?
- PrimeMcFly 3y ago
  It's an incredibly bad thing. It's also an incredibly poor excuse to justify backdooring phones. Cops need to investigate the same way they always have, look for clues, go undercover, infiltrate, find where this stuff is actually being made, etc. Scanning everyone's phones would make their jobs significantly easier, no doubt, but it simply isn't worth the cost to us as a society and there is simply no good counter-argument to that.
  - deleted 3y ago
    [deleted]
  - theshrike79 3y ago
    Let's take a step back here and bring in some facts. "Apple" wasn't scanning your phone, neither was there a "backdoor". If you would've had iCloud upload enabled (you'd be uploading all your photos to Apple's server, a place where they could scan ALL of your media anyway), the phone would've downloaded a set of hashes of KNOWN and HUMAN VERIFIED photos and videos of sexual abuse material. [1] After THREE matches of known and checked CSAM, a check done 100% on-device with zero data moving anywhere, a "reduced-quality copy" would've been sent to a human for verification. If it was someone sending you hashbombs of intentional false matches or an innocuous pic that matched because some mathematical anomaly, the actual human would notice this instantly and no action would've been taken. ...but I still think I was the only HNer who actually read Apple's spec and just didn't go with Twitter hot-takes, so I'm fighting windmills over here. Yes, there is always the risk that an authoritarian government could force Apple to insert checks for stuff other than CSAM to the downloaded database. But the exact same risk exists when you upload stuff to the cloud anyway and on an even bigger scale. (see point above about local checks not being enabled unless iCloud sync is enabled) [1] It wasn't an SHA-1 hash where changing a single bit in the source would make the hash invalid, the people doing that were actually competent.
rafale 3y ago
False positives would constitute a huge invasion of privacy. Even actual positives would be, a mom taking a private picture of her naked baby, how can you report that. They did well dropping this insane plan. The slippery slope argument is also a solid one.
- deleted 3y ago
  [deleted]
- cmcaleer 3y ago
  NYT article about exactly this situation[0]. Despite the generally technical competency of HN readership, I imagine there would be a lot of people who would find themselves completely fucked if this situation happened to them. The tl;dr is that despite this man ultimately having his name cleared by the police after having his entire Google account history (not just cloud) searched as well his logs from a warrant served to ISP, Google closed his account when the alleged CSAM was detected and never reinstated it. He lost his emails, cloud pictures, phone number (which losing access to prevented the police from contacting him via phone), and more all while going through a gross, massive invasion of his privacy because he was trying to do right for his child during a time when face-to-face doctor appointments were difficult to come by. This should be a particularly salient reminder to people to self-host at the very least the domain for their primary and professional e-mail. [0] https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html https://www.nytimes.com/2022/08/21/technology/google-surveil...
- Gigachad 3y ago
  The apple one was only matching against known images, not trying to detect new ones. The google one actually does try to detect new ones and there are reported instances of Google sending the police on normal parents for photos they took for the doctor.
  - jasonlfunk 3y ago
    And it would only notify someone for human review if a certain threshold was reached; just having one or two violating images would have tripped the system.
Simulacra 3y ago
I haven't forgot about the guy that sent photos of his child to his doctor and was investigated for child pornography. With these systems, in my humble opinion, you are just one innocent photo at the beach away from your life turned upside down.
- tjpnz 3y ago
  And Google to this day refuse to admit the mistake. They've even gone as far as to insinuate that he still is a pedo despite a police investigation clearing him.
  - dredmorbius 3y ago
    Do you have a reference on that?
- robertoandred 3y ago
  That scenario would've been impossible with Apple's system.
  - barsonme 3y ago
    It would’ve been less likely, not impossible. Perceptive hashing absolutely has issues with false positives.
    - robertoandred 3y ago
      No, it would've been impossible. One photo match wouldn't have been enough to trigger any sort of response within Apple's system. And that's ignoring the fact that his photo wouldn't have matched anyway because it isn't in any CSAM database.
      - barsonme 3y ago
        Apple’s proposal used perceptual hashing, which does have false positives, not cryptographic hashing, which does not.
_m8fo 3y ago
I’m not sure I understand Apple’s logic here. Are iCloud Photos in their data centers not scanned? Isn’t everything by default for iCloud users sent there automatically to begin with? Doesn’t the same logic around slippery slope also apply to cloud scans? This is not to say they should scan locally, but my understanding of CSAM was that it would only be scanned on its way to the cloud anyways, so users who didn’t use iCloud would’ve never been scanned to begin with. Their new proposed set of tools seems like a good enough compromise from the original proposal in any case.
- theshrike79 3y ago
  In my opinion their goal was to get stuff to a state where they could encrypt everything on iCloud so that even they can't access it. To counter the "think of the children" -argument governments use to justify surveillance, Apple tried scanning stuff on-device but the internet got a collective hissy-fit of intentionally misunderstanding the feature and it was quickly scrapped.
  - Shank 3y ago
    > In my opinion their goal was to get stuff to a state where they could encrypt everything on iCloud so that even they can't access it. They basically did. If you turn on Advanced Data Protection, you get all of the encryption benefits, sans scanning. The interesting thing is that if you turn on ADP though, binary file hashes are unencrypted on iCloud, which would theoretically allow someone to ask for those hashes in a legal request. But it's obviously not as useful for CSAM detection, as, say, PhotoDNA hashes. See: https://support.apple.com/en-us/HT202303 https://support.apple.com/en-us/HT202303
    - chatmasta 3y ago
      Nice! TIL this exists. For anyone else wondering, to enable it just go to iOS Settings -> iCloud and you'll see "Advanced Data Protection." Toggle it to enabled to create a recovery key, which you'll then be prompted to input correctly after saving it somewhere safe, and then return to the iCloud Settings page, toggle it one more time and enter your recovery key again to confirm.
gruturo 3y ago
> "Scanning every user’s privately stored iCloud data would create new threat vectors for data thieves to find and exploit" > "It would also inject the potential for a slippery slope of unintended consequences. Scanning for one type of content, for instance, opens the door for bulk surveillance and could create a desire to search other encrypted messaging systems across content types." Yes, and it was patently obvious from the onset. Why did it take a massive public backlash to actually reason about this? Can we get a promise that future initiatives will be evaluated a bit more critically before crap like this bubbles to the top again? Come on you DO hire bright people, what's your actual problem here?
- simondotau 3y ago
  Apple was concerned about governments using the excuse of CSAM to pass laws which would force Apple to weaken encryption across the board. Whether this was the right response to such concern is something I’m not unsympathetic towards. Certainly I think it’s reasonable to say that Apple was trying to thread a needle in a way which was never going to please everyone, even if it somehow turns out to have been the least-worst outcome.
  - guiambros 3y ago
    Yes, but to OP's point: this was patently obvious from the onset. Even here the comments at the time [1] pointed to all sorts of potential misuse, political or religious prosecution, dystopian cases of false positives, and that this would leave the door open to future government escalation beyond CSAM. How could they not see that they would have a giant backlash on their hands? Did they overestimate their ability to get away with the "it's for our children" excuse this badly?" [1] https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741
    - simondotau 3y ago
      I think Apple was doing the exact opposite. They wanted to do the __least possible thing__ in order to stave away the far worse outcome of intelligence departments using the "it's for our children" excuse to pressure elected representatives to vote for back doors on consumer encryption. The ridiculous thing is that Apple's proposal was functionally identical to what other platform vendors (e.g. Google, Microsoft) were already doing. In all cases — including Apple's proposed system — only photos uploaded to cloud storage would be scanned to see if it matched CSAM already known to the government. The only difference with Apple's proposal was initial "fuzzy hash" calculation would be performed on-device prior to upload, instead of on-cloud after upload. The reason for doing it differently was because it meant (in theory) satisfying both masters — implementing real end-to-end encryption, while not being seen as a CSAM scanning laggard compared to Google, Microsoft, etc. Other vendors just scan all your shit and nobody cares.
sneak 3y ago
The vast majority (99%+) of iCloud Photos are not e2ee and are readable to Apple. You can rest assured that they are scanning all of it serverside for illegal images presently. The kerfuffle was around clientside scanning, something that it has been reported that they dropped. I have thus far seen no statements from Apple that they actually intended to stop the deployment of clientside scanning. Serverside scanning has been possible (and likely) for a long time, which illuminates their "slippery slope" argument as farce (unless they intend to force migrate everyone to e2ee storage in the future).
- h1fra 3y ago
  Where do you get this number?
  - sneak 3y ago
    e2ee for iCloud is currently opt-in, without prompts/nudging. Most power users don't even have it turned on or are aware of its existence. The setting is buried/hidden in submenus. Approximately no one uses it. Hopefully Apple will begin promoting users to migrate in future updates.
    - 0x000042 3y ago
      > The setting is buried/hidden in submenus. Mind sharing where it is on an iPhone and Mac? I have not been able to find it.
      - smilespray 3y ago
        How to turn on Advanced Data Protection for iCloud https://support.apple.com/en-gb/HT212520 https://support.apple.com/en-gb/HT212520
      - michaelt 3y ago
        It's called "Advanced Data Protection for iCloud" It's kinda complicated to turn on, as it disables most account recovery options.
deleted 3y ago
[deleted]
menzoic 3y ago
Pretty ridiculous idea. Bad actors simply won't use their platform if this was in place. It would only be scanning private data from all people who aren't comitting crimes.
- deleted 3y ago
  [deleted]
- stuartjohnson12 3y ago
  You'd be surprised. Lots of offenders are very low sophistication. If you read news articles about how a particular offender was caught with illegal material, so so often it's because they uploaded it to a cloud provider. It's not a one-sided tradeoff here.
  - akira2501 3y ago
    What percentage of offenders victimize children and never record it in any way? If that's the overwhelming majority of abuse cases, what are we even doing here?
    - seanieb 3y ago
      Worse. Only a tiny fraction of child abuse material cases actually get investigated due to lack of resources… this debate about scanning is an insane distraction.
    - stuartjohnson12 3y ago
      Policing doesn't stop all crime, why are we even policing?
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
baz00 3y ago
It's nice that Apple have clarified this. I think that the original intent was a misstep and possibly an internal political situation that they had to deal with. I can see that a number of people would be on each side of the debate with advocacy throughout the org. There is only one correct answer though and that is what they have clarified. I would immediately leave the platform if they progressed with this.
jmyeet 3y ago
Part of the reason why this was (and is) a terrible idea is how these companies operate and the cost and stigma of a false negative. Companies don't want to employ people. People are annoying. They make annoying demands like wanting time off and having enough money to not be homeless or starving. AI should be a tool that enhances the productivity of a worker rather than replacing them. Fully automated "safety" systems always get weaponized. This is really apparent on Tiktok where reporting users you don't like is clearly brigaded becasue a certain number of reports in a given period triggers automatic takedowns and bans regardless of assurances there is human review (there isn't). It's so incredibly obvious when you see a duet with a threatening video gets taken down while the original video doesn't (with reports showing "No violation"). Additionally, companies like to just ban your account with absolutely no explanation, accountability, right to review or right to appeal. Again, all those things would require employing people. False positives can be incredibly damaging. Not only could this result in your account being banned (possibly with the loss of all your photos on something like iCloud/iPhotos) but it may get you in trouble with law enforcement. Don't believe me? Hertz was falsely reported their cars being stolen [1], which created massive problems for those affected. In a better world, Hertz executives would be in prison for making false police reports (which, for you and me, is a crime) but that will never happen to executives. It still requires human review to identify offending content. Mass shootings have been live streamed. No automatic system is going to be able to accurately differentiate between this and, say, a movie scene. I guarantee you any automated system will have similar problems differentiating between actual CSAM and, say, a child in the bath or at the beach. These companies don't want to solve these problems. They simply want legal and PR cover for appearing to solve them, consequences be damned. [1]: https://www.npr.org/2022/12/06/1140998674/hertz-false-accusation-stealing-cars-settlement https://www.npr.org/2022/12/06/1140998674/hertz-false-accusa...
- robertoandred 3y ago
  Good thing this wasn't fully automated and there would've been human review. The whole uproar about this system was made by people who didn't know the most basic things about it.
beej71 3y ago
I think they likely also considered the lawsuit exposure. If just 0.0001% of users sued over false positives, Apple would be in serious trouble. And there's another dynamic where telling your customers you're going to scan their content for child porn is the same as saying you suspect your customers of having child porn. And your average non-criminal customer's reaction to that is not positive for multiple reasons.
- advisedwang 3y ago
  Section 230 removes liability for restricting good faith attempts to combat CSAM. > (2) Civil liability > No provider or user of an interactive computer service shall be held liable on account of— > (A) any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected
  - nottorp 3y ago
    I don't see any reference to child porn there. Who decides what's obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable? Especially otherwise objectionable?
    - stale2002 3y ago
      > Who decides Judges and juries. And I can assure you that every single judge in the USA, and almost every single member of a jury would decide that CSAM is obscene. Thats how the law works. We have tons of laws that use general words like this, and trying to be "clever" usually just results in a lost court case or prison time for the person who thinks they found a loophole.
      - nottorp 3y ago
        What else is obscene? And why would I have to accept an US jury's opinion? Also you didn't define 'otherwise objectionable'. For example what I think is excessive violence may be considered normal by a jury in the US...
neonate 3y ago
http://web.archive.org/web/20230901190025/https://www.wired.com/story/apple-csam-scanning-heat-initiative-letter/ http://web.archive.org/web/20230901190025/https://www.wired.... https://archive.ph/HZVdd https://archive.ph/HZVdd
MBlume 3y ago
The article keeps saying that Apple has responded or that Apple has clarified and then linking other Wired articles. Is there an Apple press release somewhere? If so, I'd rather read that. ETA: looks like they directly provide documents from Apple at the bottom of the article
- TheHappyOddish 3y ago
  Here's the link: https://s3.documentcloud.org/documents/23933180/apple-letter-to-heat-initiative.pdf https://s3.documentcloud.org/documents/23933180/apple-letter...
tzs 3y ago
I'm curious about the new parental control features they announced at the same time as the iCloud photo scanning. My recollection is that when they withdrew the iCloud scanning they also withdrew the new parental controls. I'm curious why they also withdrew those. For those who don't remember the parental control, which were largely overshadowed by the controversy over the cloud stuff, they were to work like this: 1. If parents had enabled them on their child's device, they would scan incoming messages for sexual material. The scan would be entirely on-device. If such material was found the material would be blocked, the child would be notified that the the message contained material that their parents thought might be harmful, and asked if they wanted to see it anyway. 2. If the child said no, the material would be dropped and that would be the end of it. If the child said yes what happened next depended on the age of the child. 3. If the was at least 13 years old the material would be unblocked and that would be the end of it. 4. If the while was not yet 13 they would be given another warning that their parents think the material might be harmful, and again asked if they want to go ahead and see it. They would be told that if they say "yes" their parents will be notified that they viewed the material. 5. If they say no the material remains blocked and that is the end of it. 6. If they say yes it is unblocked, but their parents are told. There wasn't a lot of discussion of this, and I only recall seeing one major privacy group object (the EFF, on the grounds that if it reaches step 6 it violates the privacy of the person sending sex stuff to your pre-teen because they probably did not intend for the parents to know).
- Shank 3y ago
  The issue is that it’s predicated on an age field that can be set separately. It’s easy to use parental controls to control non-children by setting a lower age internally. Think victims of human trafficking or adults in odd relationship situations. Not quite the same but see: https://www.forbes.com/sites/thomasbrewster/2023/04/06/sex-traffickers-use-parenting-apps-like-life360-to-spy-on-victims/ https://www.forbes.com/sites/thomasbrewster/2023/04/06/sex-t... Apple’s updated system allows children to ask for help from an adult using the Communication Safety features, which strikes a good balance to me.
  - nyolfen 3y ago
smittywerben 3y ago
It surprises me that people think their common $CLOUD_STORAGE_PROVIDER doesn't already scan files for child exploitation material.
SergeAx 3y ago
Speculation: they did a trial on random accounts from all over the world and found out so much illegal content that it will make them do enormous amount of policing on scale and lose troves of customers.
ogurechny 3y ago
> Child sexual abuse material is abhorrent and we are committed to breaking the chain of coercion and influence that makes children susceptible to it. It is amazing that so much counter-cultural spirit remains in Apple. They are probably going to ban likes and other vanity features in all iOS applications, prohibit access to popular media, put “pop stars” into rehabs, and teach their users to disobey (the hardest of all tasks). A lot of people try really hard not to see that “unusual” abuse of the children is the same as “usual” abuse of everyone. Conveniently, the need for distinction creates “maniacs” that are totally, totally different from “normal people”, and cranks up the sensation level. The discussion of “external” evil then can continue ad infinitum without dealing with status quo of “peaceful, normal life”.
erostrate 3y ago
Sarah Gardner, the author of the letter to Apple and CEO of the Heat Initiative, worked for 10 years and until earlier this year as a VP at Thorn [1]. Thorn sells a "comprehensive solution for platforms to identify, remove and report child sexual abuse material." [2] She's using PR to pressure on Apple into implementing the kind of solution her previous company is selling. Won't someone think of the children?? [1] https://www.linkedin.com/in/sarah-gardner-aba90013/ https://www.linkedin.com/in/sarah-gardner-aba90013/ [2] https://www.thorn.org/our-work-to-stop-child-sexual-exploitation/ https://www.thorn.org/our-work-to-stop-child-sexual-exploita...

224 points by Anonboxis 3y ago

_boffin_ 3y ago

So… they’re just doing on device scanning instead of icloud and just calling it a different name?

meepmorp 3y ago
Not really? It looks like the nudity detection features are all on device, aren't CSAM specific, and seem to be mostly geared towards blocking stuff like unsolicited dick pics. The earlier design was a hybrid model that scanned for CSAM on device, then flagged files were reviewed on upload.
olliej 3y ago
No, the terrible misfeature that this group wants is “government provides a bunch of opaque hashes that are ‘CSAM’, all images are compared with those hashes, and if the hashes match then the user details are given to police” Note that by design the hashes cannot be audited (though in the legitimate case I don’t imagine doing so would be pleasant), so there’s nothing stopping a malicious party inserting hashes of anything they want - and then the news report will be “person x bought in for questioning after CSAM detector flagged them”. That’s before countries just pass explicit laws saying that the filter must includE LGBT content (in the US several states consider books with lgbt characters to be sexual content, so a lgbt teenager would be de facto CSAM), in the UK the IPA is used to catch people not collecting dog poop so trusting them not to expand scope is laughable, in Iran a picture of a woman without a hijab would obviously be reportable, etc What Apple has done is add the ability to filter content (eg block dick picks) and for child accounts to place extra steps (incl providing contact numbers I think?) if a child attempts to send pics with nudity, etc
- Dig1t 3y ago
  >in the UK the IPA is used to catch people not collecting dog poop What does this mean? What is IPA? I tried Googling for it but I’m not finding much. I would love to learn more about that
  - olliej 3y ago
    The investigatory powers act. It was passed to stop terrorism, because previously they found that having multiple people (friends and family etc) report that someone was planning a terrorist attack failed to stop a terrorist attack.
- robertoandred 3y ago
  How exactly would you be able to "filter" LGBT content? I don't think you understand how this system would've worked.
  - olliej 3y ago
    Hypothetically you have hashes for two people of gender X (lets be honest, based on popularity of different types of porn two men). This is not meaningfully different from opaque hash of "CSAM". But you're missing the point: Step 1. generate some opaque hash of the "semantics" of an image Step 2. compare those hashes to some list of hashes of "CSAM", which again fundamentally cannot be audited Step 3. report any hits to law enforcement Step 4. person X is being investigated due to reported violations of laws against child abuse. Basically: how do you design a system in which the state provides "semantic" hashes of "CSAM" that cannot be trivially abused by inclusion of non-CSAM as "CSAM", or by laws mandating inclusion of things that are objectively not-CSAM. Hypothetically: hashes that match christian crosses, star of David, muslim star and/or crescent, etc. Or in the US DNC, RNC, pride, etc flags. Recall that definitionally no one can audit the hashes that would trigger notifying law encforcement.
    - robertoandred 3y ago
      Except this system wouldn't have looked at "semantics". You can't simply match a hash of a cross or star or flag, you have to match a specific photograph. Which photograph do you use?
sheepscreek 3y ago
Yes - and there’s a huge difference between the two. In a word, decentralization. By detecting unsafe material on-device / while it is being created, they can prevent it from being shared. And because this happens on individual devices, Apple doesn’t need to know what’s on people’s iCloud. So they can offer end-to-end encryption, where even the data on their servers is encrypted. Only your devices can “see” it (it’s a black box for Apple servers, gibberish - without the correct decryption key).

formerly_proven 3y ago

The who is often interesting with these stories. > a new child safety group known as Heat Initiative Doesn't even have a website or any kind of social media presence; it literally doesn't appear to exist apart from the reporting on Apple's response to them, which is entirely based on Apple sharing their response with media, not the group interacting with media. > Sarah Gardner on the other hand previously appeared as the VP of External Affairs (i.e. Marketing) of Thorn (formerly DNA Foundation): https://www.thorn.org/blog/searching-for-a-child-in-a-private-world-thorn-vp-of-external-affairs-speaks-at-tedxwarwick/ https://www.thorn.org/blog/searching-for-a-child-in-a-privat... So despite looking a bit fishy at first, this doesn't seem to come from a christofascist group.

figlett 3y ago
> So despite looking a bit fishy at first, this doesn't seem to come from a christofascist group. Why would you assume this in the first place?
- bsenftner 3y ago
  They use hysteria to generate power in society.
- krapp 3y ago
  The main impetus behind "child safety" advocacy nowadays seem to be by cells of extremist right-wing Christian / QAnon types who believe in conspiracy theories like Pizzagate and the "gay groomer" panic. It's a reasonable assumption to make about any such group mentioned in the media that doesn't have an established history at least prior to 2016.
  - figlett 3y ago
    It sounds like an entirely unreasonable assumption to me. Advocating for child safety is something that transcends political differences, and generally unifies people across the political spectrum. I mean, there aren't many people who want paedophiles to be able to amass huge collections of child abuse imagery from other paedophiles online. And pretty much every parent wants their child to be kept safe from predators both online and offline.

gnfargbl 3y ago

> “Scanning every user’s privately stored iCloud data would create new threat vectors for data thieves to find and exploit," Neuenschwander wrote. "It would also inject the potential for a slippery slope of unintended consequences. Scanning for one type of content, for instance, opens the door for bulk surveillance and could create a desire to search other encrypted messaging systems across content types.” Both of these arguments are absolutely, unambiguously, correct. The other side of the coin is that criminals are using E2EE communication systems to share sexual abuse material in ways and at rates which they were not previously able to. This is, I argue, a bad thing. Is is bad for the individuals who are re-victimised on every share. It is also bad for the fabric of society at large, in the sense that if we don't clearly take a stand against abhorrent behaviour then we are in some sense condoning it. Does the tech industry have any alternate solutions that could functionally mitigate this abuse? Does the industry feel that it has any responsibility at all to do so? Or do we all just shout "yay, individual freedom wins again!" and forget about the actual problem that this (misguided) initiative was originally aimed at?

PrimeMcFly 3y ago
It's an incredibly bad thing. It's also an incredibly poor excuse to justify backdooring phones. Cops need to investigate the same way they always have, look for clues, go undercover, infiltrate, find where this stuff is actually being made, etc. Scanning everyone's phones would make their jobs significantly easier, no doubt, but it simply isn't worth the cost to us as a society and there is simply no good counter-argument to that.
- deleted 3y ago
  [deleted]
- theshrike79 3y ago
  Let's take a step back here and bring in some facts. "Apple" wasn't scanning your phone, neither was there a "backdoor". If you would've had iCloud upload enabled (you'd be uploading all your photos to Apple's server, a place where they could scan ALL of your media anyway), the phone would've downloaded a set of hashes of KNOWN and HUMAN VERIFIED photos and videos of sexual abuse material. [1] After THREE matches of known and checked CSAM, a check done 100% on-device with zero data moving anywhere, a "reduced-quality copy" would've been sent to a human for verification. If it was someone sending you hashbombs of intentional false matches or an innocuous pic that matched because some mathematical anomaly, the actual human would notice this instantly and no action would've been taken. ...but I still think I was the only HNer who actually read Apple's spec and just didn't go with Twitter hot-takes, so I'm fighting windmills over here. Yes, there is always the risk that an authoritarian government could force Apple to insert checks for stuff other than CSAM to the downloaded database. But the exact same risk exists when you upload stuff to the cloud anyway and on an even bigger scale. (see point above about local checks not being enabled unless iCloud sync is enabled) [1] It wasn't an SHA-1 hash where changing a single bit in the source would make the hash invalid, the people doing that were actually competent.

rafale 3y ago

False positives would constitute a huge invasion of privacy. Even actual positives would be, a mom taking a private picture of her naked baby, how can you report that. They did well dropping this insane plan. The slippery slope argument is also a solid one.

deleted 3y ago
[deleted]
cmcaleer 3y ago
NYT article about exactly this situation[0]. Despite the generally technical competency of HN readership, I imagine there would be a lot of people who would find themselves completely fucked if this situation happened to them. The tl;dr is that despite this man ultimately having his name cleared by the police after having his entire Google account history (not just cloud) searched as well his logs from a warrant served to ISP, Google closed his account when the alleged CSAM was detected and never reinstated it. He lost his emails, cloud pictures, phone number (which losing access to prevented the police from contacting him via phone), and more all while going through a gross, massive invasion of his privacy because he was trying to do right for his child during a time when face-to-face doctor appointments were difficult to come by. This should be a particularly salient reminder to people to self-host at the very least the domain for their primary and professional e-mail. [0] https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html https://www.nytimes.com/2022/08/21/technology/google-surveil...
Gigachad 3y ago
The apple one was only matching against known images, not trying to detect new ones. The google one actually does try to detect new ones and there are reported instances of Google sending the police on normal parents for photos they took for the doctor.
- jasonlfunk 3y ago
  And it would only notify someone for human review if a certain threshold was reached; just having one or two violating images would have tripped the system.

Simulacra 3y ago

I haven't forgot about the guy that sent photos of his child to his doctor and was investigated for child pornography. With these systems, in my humble opinion, you are just one innocent photo at the beach away from your life turned upside down.

tjpnz 3y ago
And Google to this day refuse to admit the mistake. They've even gone as far as to insinuate that he still is a pedo despite a police investigation clearing him.
- dredmorbius 3y ago
  Do you have a reference on that?
robertoandred 3y ago
That scenario would've been impossible with Apple's system.
- barsonme 3y ago
  It would’ve been less likely, not impossible. Perceptive hashing absolutely has issues with false positives.
  - robertoandred 3y ago
    No, it would've been impossible. One photo match wouldn't have been enough to trigger any sort of response within Apple's system. And that's ignoring the fact that his photo wouldn't have matched anyway because it isn't in any CSAM database.
    - barsonme 3y ago
      Apple’s proposal used perceptual hashing, which does have false positives, not cryptographic hashing, which does not.

_m8fo 3y ago

I’m not sure I understand Apple’s logic here. Are iCloud Photos in their data centers not scanned? Isn’t everything by default for iCloud users sent there automatically to begin with? Doesn’t the same logic around slippery slope also apply to cloud scans? This is not to say they should scan locally, but my understanding of CSAM was that it would only be scanned on its way to the cloud anyways, so users who didn’t use iCloud would’ve never been scanned to begin with. Their new proposed set of tools seems like a good enough compromise from the original proposal in any case.

theshrike79 3y ago
In my opinion their goal was to get stuff to a state where they could encrypt everything on iCloud so that even they can't access it. To counter the "think of the children" -argument governments use to justify surveillance, Apple tried scanning stuff on-device but the internet got a collective hissy-fit of intentionally misunderstanding the feature and it was quickly scrapped.
- Shank 3y ago
  > In my opinion their goal was to get stuff to a state where they could encrypt everything on iCloud so that even they can't access it. They basically did. If you turn on Advanced Data Protection, you get all of the encryption benefits, sans scanning. The interesting thing is that if you turn on ADP though, binary file hashes are unencrypted on iCloud, which would theoretically allow someone to ask for those hashes in a legal request. But it's obviously not as useful for CSAM detection, as, say, PhotoDNA hashes. See: https://support.apple.com/en-us/HT202303 https://support.apple.com/en-us/HT202303
  - chatmasta 3y ago
    Nice! TIL this exists. For anyone else wondering, to enable it just go to iOS Settings -> iCloud and you'll see "Advanced Data Protection." Toggle it to enabled to create a recovery key, which you'll then be prompted to input correctly after saving it somewhere safe, and then return to the iCloud Settings page, toggle it one more time and enter your recovery key again to confirm.

gruturo 3y ago

> "Scanning every user’s privately stored iCloud data would create new threat vectors for data thieves to find and exploit" > "It would also inject the potential for a slippery slope of unintended consequences. Scanning for one type of content, for instance, opens the door for bulk surveillance and could create a desire to search other encrypted messaging systems across content types." Yes, and it was patently obvious from the onset. Why did it take a massive public backlash to actually reason about this? Can we get a promise that future initiatives will be evaluated a bit more critically before crap like this bubbles to the top again? Come on you DO hire bright people, what's your actual problem here?

simondotau 3y ago
Apple was concerned about governments using the excuse of CSAM to pass laws which would force Apple to weaken encryption across the board. Whether this was the right response to such concern is something I’m not unsympathetic towards. Certainly I think it’s reasonable to say that Apple was trying to thread a needle in a way which was never going to please everyone, even if it somehow turns out to have been the least-worst outcome.
- guiambros 3y ago
  Yes, but to OP's point: this was patently obvious from the onset. Even here the comments at the time [1] pointed to all sorts of potential misuse, political or religious prosecution, dystopian cases of false positives, and that this would leave the door open to future government escalation beyond CSAM. How could they not see that they would have a giant backlash on their hands? Did they overestimate their ability to get away with the "it's for our children" excuse this badly?" [1] https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741
  - simondotau 3y ago
    I think Apple was doing the exact opposite. They wanted to do the __least possible thing__ in order to stave away the far worse outcome of intelligence departments using the "it's for our children" excuse to pressure elected representatives to vote for back doors on consumer encryption. The ridiculous thing is that Apple's proposal was functionally identical to what other platform vendors (e.g. Google, Microsoft) were already doing. In all cases — including Apple's proposed system — only photos uploaded to cloud storage would be scanned to see if it matched CSAM already known to the government. The only difference with Apple's proposal was initial "fuzzy hash" calculation would be performed on-device prior to upload, instead of on-cloud after upload. The reason for doing it differently was because it meant (in theory) satisfying both masters — implementing real end-to-end encryption, while not being seen as a CSAM scanning laggard compared to Google, Microsoft, etc. Other vendors just scan all your shit and nobody cares.

sneak 3y ago

The vast majority (99%+) of iCloud Photos are not e2ee and are readable to Apple. You can rest assured that they are scanning all of it serverside for illegal images presently. The kerfuffle was around clientside scanning, something that it has been reported that they dropped. I have thus far seen no statements from Apple that they actually intended to stop the deployment of clientside scanning. Serverside scanning has been possible (and likely) for a long time, which illuminates their "slippery slope" argument as farce (unless they intend to force migrate everyone to e2ee storage in the future).

h1fra 3y ago
Where do you get this number?
- sneak 3y ago
  e2ee for iCloud is currently opt-in, without prompts/nudging. Most power users don't even have it turned on or are aware of its existence. The setting is buried/hidden in submenus. Approximately no one uses it. Hopefully Apple will begin promoting users to migrate in future updates.
  - 0x000042 3y ago
    > The setting is buried/hidden in submenus. Mind sharing where it is on an iPhone and Mac? I have not been able to find it.
    - smilespray 3y ago
      How to turn on Advanced Data Protection for iCloud https://support.apple.com/en-gb/HT212520 https://support.apple.com/en-gb/HT212520
    - michaelt 3y ago
      It's called "Advanced Data Protection for iCloud" It's kinda complicated to turn on, as it disables most account recovery options.

deleted 3y ago

[deleted]

menzoic 3y ago

Pretty ridiculous idea. Bad actors simply won't use their platform if this was in place. It would only be scanning private data from all people who aren't comitting crimes.

deleted 3y ago
[deleted]
stuartjohnson12 3y ago
You'd be surprised. Lots of offenders are very low sophistication. If you read news articles about how a particular offender was caught with illegal material, so so often it's because they uploaded it to a cloud provider. It's not a one-sided tradeoff here.
- akira2501 3y ago
  What percentage of offenders victimize children and never record it in any way? If that's the overwhelming majority of abuse cases, what are we even doing here?
  - seanieb 3y ago
    Worse. Only a tiny fraction of child abuse material cases actually get investigated due to lack of resources… this debate about scanning is an insane distraction.
  - stuartjohnson12 3y ago
    Policing doesn't stop all crime, why are we even policing?
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

baz00 3y ago

It's nice that Apple have clarified this. I think that the original intent was a misstep and possibly an internal political situation that they had to deal with. I can see that a number of people would be on each side of the debate with advocacy throughout the org. There is only one correct answer though and that is what they have clarified. I would immediately leave the platform if they progressed with this.

jmyeet 3y ago

Part of the reason why this was (and is) a terrible idea is how these companies operate and the cost and stigma of a false negative. Companies don't want to employ people. People are annoying. They make annoying demands like wanting time off and having enough money to not be homeless or starving. AI should be a tool that enhances the productivity of a worker rather than replacing them. Fully automated "safety" systems always get weaponized. This is really apparent on Tiktok where reporting users you don't like is clearly brigaded becasue a certain number of reports in a given period triggers automatic takedowns and bans regardless of assurances there is human review (there isn't). It's so incredibly obvious when you see a duet with a threatening video gets taken down while the original video doesn't (with reports showing "No violation"). Additionally, companies like to just ban your account with absolutely no explanation, accountability, right to review or right to appeal. Again, all those things would require employing people. False positives can be incredibly damaging. Not only could this result in your account being banned (possibly with the loss of all your photos on something like iCloud/iPhotos) but it may get you in trouble with law enforcement. Don't believe me? Hertz was falsely reported their cars being stolen [1], which created massive problems for those affected. In a better world, Hertz executives would be in prison for making false police reports (which, for you and me, is a crime) but that will never happen to executives. It still requires human review to identify offending content. Mass shootings have been live streamed. No automatic system is going to be able to accurately differentiate between this and, say, a movie scene. I guarantee you any automated system will have similar problems differentiating between actual CSAM and, say, a child in the bath or at the beach. These companies don't want to solve these problems. They simply want legal and PR cover for appearing to solve them, consequences be damned. [1]: https://www.npr.org/2022/12/06/1140998674/hertz-false-accusation-stealing-cars-settlement https://www.npr.org/2022/12/06/1140998674/hertz-false-accusa...

robertoandred 3y ago
Good thing this wasn't fully automated and there would've been human review. The whole uproar about this system was made by people who didn't know the most basic things about it.

beej71 3y ago

I think they likely also considered the lawsuit exposure. If just 0.0001% of users sued over false positives, Apple would be in serious trouble. And there's another dynamic where telling your customers you're going to scan their content for child porn is the same as saying you suspect your customers of having child porn. And your average non-criminal customer's reaction to that is not positive for multiple reasons.

advisedwang 3y ago
Section 230 removes liability for restricting good faith attempts to combat CSAM. > (2) Civil liability > No provider or user of an interactive computer service shall be held liable on account of— > (A) any action voluntarily taken in good faith to restrict access to or availability of material that the provider or user considers to be obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable, whether or not such material is constitutionally protected
- nottorp 3y ago
  I don't see any reference to child porn there. Who decides what's obscene, lewd, lascivious, filthy, excessively violent, harassing, or otherwise objectionable? Especially otherwise objectionable?
  - stale2002 3y ago
    > Who decides Judges and juries. And I can assure you that every single judge in the USA, and almost every single member of a jury would decide that CSAM is obscene. Thats how the law works. We have tons of laws that use general words like this, and trying to be "clever" usually just results in a lost court case or prison time for the person who thinks they found a loophole.
    - nottorp 3y ago
      What else is obscene? And why would I have to accept an US jury's opinion? Also you didn't define 'otherwise objectionable'. For example what I think is excessive violence may be considered normal by a jury in the US...

neonate 3y ago

http://web.archive.org/web/20230901190025/https://www.wired.com/story/apple-csam-scanning-heat-initiative-letter/ http://web.archive.org/web/20230901190025/https://www.wired.... https://archive.ph/HZVdd https://archive.ph/HZVdd

MBlume 3y ago

The article keeps saying that Apple has responded or that Apple has clarified and then linking other Wired articles. Is there an Apple press release somewhere? If so, I'd rather read that. ETA: looks like they directly provide documents from Apple at the bottom of the article

TheHappyOddish 3y ago
Here's the link: https://s3.documentcloud.org/documents/23933180/apple-letter-to-heat-initiative.pdf https://s3.documentcloud.org/documents/23933180/apple-letter...

tzs 3y ago

I'm curious about the new parental control features they announced at the same time as the iCloud photo scanning. My recollection is that when they withdrew the iCloud scanning they also withdrew the new parental controls. I'm curious why they also withdrew those. For those who don't remember the parental control, which were largely overshadowed by the controversy over the cloud stuff, they were to work like this: 1. If parents had enabled them on their child's device, they would scan incoming messages for sexual material. The scan would be entirely on-device. If such material was found the material would be blocked, the child would be notified that the the message contained material that their parents thought might be harmful, and asked if they wanted to see it anyway. 2. If the child said no, the material would be dropped and that would be the end of it. If the child said yes what happened next depended on the age of the child. 3. If the was at least 13 years old the material would be unblocked and that would be the end of it. 4. If the while was not yet 13 they would be given another warning that their parents think the material might be harmful, and again asked if they want to go ahead and see it. They would be told that if they say "yes" their parents will be notified that they viewed the material. 5. If they say no the material remains blocked and that is the end of it. 6. If they say yes it is unblocked, but their parents are told. There wasn't a lot of discussion of this, and I only recall seeing one major privacy group object (the EFF, on the grounds that if it reaches step 6 it violates the privacy of the person sending sex stuff to your pre-teen because they probably did not intend for the parents to know).

Shank 3y ago
The issue is that it’s predicated on an age field that can be set separately. It’s easy to use parental controls to control non-children by setting a lower age internally. Think victims of human trafficking or adults in odd relationship situations. Not quite the same but see: https://www.forbes.com/sites/thomasbrewster/2023/04/06/sex-traffickers-use-parenting-apps-like-life360-to-spy-on-victims/ https://www.forbes.com/sites/thomasbrewster/2023/04/06/sex-t... Apple’s updated system allows children to ask for help from an adult using the Communication Safety features, which strikes a good balance to me.
- nyolfen 3y ago

smittywerben 3y ago

It surprises me that people think their common $CLOUD_STORAGE_PROVIDER doesn't already scan files for child exploitation material.

SergeAx 3y ago

Speculation: they did a trial on random accounts from all over the world and found out so much illegal content that it will make them do enormous amount of policing on scale and lose troves of customers.

ogurechny 3y ago

> Child sexual abuse material is abhorrent and we are committed to breaking the chain of coercion and influence that makes children susceptible to it. It is amazing that so much counter-cultural spirit remains in Apple. They are probably going to ban likes and other vanity features in all iOS applications, prohibit access to popular media, put “pop stars” into rehabs, and teach their users to disobey (the hardest of all tasks). A lot of people try really hard not to see that “unusual” abuse of the children is the same as “usual” abuse of everyone. Conveniently, the need for distinction creates “maniacs” that are totally, totally different from “normal people”, and cranks up the sensation level. The discussion of “external” evil then can continue ad infinitum without dealing with status quo of “peaceful, normal life”.

erostrate 3y ago

Sarah Gardner, the author of the letter to Apple and CEO of the Heat Initiative, worked for 10 years and until earlier this year as a VP at Thorn [1]. Thorn sells a "comprehensive solution for platforms to identify, remove and report child sexual abuse material." [2] She's using PR to pressure on Apple into implementing the kind of solution her previous company is selling. Won't someone think of the children?? [1] https://www.linkedin.com/in/sarah-gardner-aba90013/ https://www.linkedin.com/in/sarah-gardner-aba90013/ [2] https://www.thorn.org/our-work-to-stop-child-sexual-exploitation/ https://www.thorn.org/our-work-to-stop-child-sexual-exploita...