3 ms·

> It is crazy people think apple isnt on the side of privacy. > It also ensured pressure from governments and plaintiffs, including CSAM victims, who preferred

by avidiax 2mo ago

> It is crazy people think apple isnt on the side of privacy. > It also ensured pressure from governments and plaintiffs, including CSAM victims, who preferred Apple’s more interventionist approaches, which Apple had voluntarily demonstrated it was willing to do. I feel that Apple open pandora's box with the client-side scanning. It proved that it was technically feasible, and was "privacy preserving". I use scare quotes there because I don't think that political or religious dissidents would find that the same or similar technology used to discover and persecute them is "privacy preserving". And that's really the problem with Apple here. They provided a model for scanning for any kind of message or material while purportedly maintaining privacy.

mmmlinux 2mo ago
Is it different than telling your therapist something in confidence and then finding police waiting for you in the lobby.
- arcticbull 2mo ago
  Yes, in the sense that you have a legal doctor-patient privilege that binds what they can share with whom. There's not really an Apple cloud user privilege. No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.
  - busterarm 2mo ago
    > No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc. And therapists are legally mandated to report you if you told them you viewed or possessed CSAM.
    - Dylan16807 2mo ago
      No matter how or why? That seems like a terrible mandate.
      - busterarm 2mo ago
        > No matter how or why? That seems like a terrible mandate. Honestly shocked that anyone would even say this, but even giving you the benefit of the doubt here -- the one case where I could imagine this might not happen would be if you're a police officer investigating such cases. But they also have their own therapists dedicated/trained in police-specific issues.
        Dylan16807 2mo ago
        Even if someone went browsing for it, yes that's illegal but there's no benefit in their therapist reporting them for just visiting terrible websites. But also there are definitely ways to get accidentally exposed. That's an absolutely awful thing to call the cops over.
        busterarm 2mo ago
        I think you're demonstrating incredibly poor judgment here. CSAM is a crime with real victims. Even if your patient came across it innocently, someone is out there intentionally distributing it and that needs to be investigated.
        AnthonyMouse 2mo ago
        Have you considered the implications of what you're saying? A whistleblower goes to a therapist, stressed out over their pending decision to reveal official misconduct. They've been investigating ways to post something on the internet that can't be immediately taken down by the corrupt government officials they want to expose. They express their discomfort, in confidence, to their therapist, about using something they've discovered is also used for CSAM. You think it's a good thing for the therapist to be required to report this? Should they report that the patient admitted to viewing CSAM with no context so the whistleblower gets investigated and arrested, or should they provide the context -- that the patient is about to expose the corruption of the government receiving the report? For that matter, consider what it does when someone is actually a pedophile. They find out that if they try to seek therapy to address their perverse attraction to kids, the therapist isn't allowed to keep their confidence and they'll be arrested, so instead of seeking professional help, they keep abusing kids. Is that the result we wanted? There is a reason doctor-patient confidentiality was a thing.
    - freehorse 2mo ago
      That's not generally true. From [0] > Across most states, viewing CSEM alone is generally not a mandated-reporting trigger; reporting becomes obligatory when disclosures involve an identifiable child being abused or used to produce material. > California’s CANRA imposes a distinct duty to report electronic access (download/stream) with identifying patient information, upheld against privacy challenges based on compelling state interest. [0] https://www.psychiatrictimes.com/view/mandatory-reporting-child-sexual-exploitation-material https://www.psychiatrictimes.com/view/mandatory-reporting-ch...
  - AlexandrB 2mo ago
    A better analogy is a storage locker. AFAIK police need a warrant to search "your" storage locker even though it's on someone else's property. I don't see why data in the cloud should be any different. Pre-emptively scanning everyone's data is equivalent to officers rummaging through all the storage lockers in a facility "just in case" they find something illegal.
    - d1sxeyes 2mo ago
      It’s a bit more like requiring you to submit to a weapons pat down before you go to your locker I think.
bayindirh 2mo ago
> It proved that it was technically feasible, and was "privacy preserving". Didn't their paper disproved by reversing the perceptual hashes to reveal blurred version of the images being hashed, and Apple basically said "that's fair, it's not as robust as we wanted, let's visit this later"? If not, I'll happily stand corrected, but please share sources. Addenda: - Apple's original paper: https://web.archive.org/web/20210807165030/https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://web.archive.org/web/20210807165030/https://www.apple... - Paper breaking the hash: https://arxiv.org/abs/2111.06628 https://arxiv.org/abs/2111.06628 Edit: The second one is the wrong paper. I’ll find and link the correct one tomorrow. Keeping the link for transparency.
- yogorenapan 2mo ago
  > to reveal blurred version of the images being hashed Skimmed your linked paper. It seems they were able to classify hashes up to ~8% top-1 accuracy and ~30% top-10. Not exactly a blurred version, or any images at all. So for example, they can say that you probably have images of trees, or images of buildings, but without much other data & very low accuracy. I'd still be a lot more concerned about them simply flagging political images rather than trying to get a broad understanding of what type of photos I have
  - soulofmischief 2mo ago
    It starts at a broad understanding and ends with people permanently giving up their right and ability to keep rogue corporate governments in check.
- comex 2mo ago
  The paper you linked doesn’t reveal blurred versions of the images being hashed. It does train a classifier to determine which of 1,000 ImageNet classes an image belongs to, which “achieved a top-1 test accuracy of 4.34%”.
  - bayindirh 2mo ago
    Then, that’s the wrong paper. I’ll find it and link it as a reply to this comment. Probably tomorrow morning.
trollbridge 2mo ago
I thought the client side scanning was to protect children? If it suspects an image is bad, it blurs it and pops up a warning including a link to resources to go to for help. Very different than trying to narc out users to the authorities.
- pavon 2mo ago
  That is what they actually deployed. They were planning on performing client-side scanning of all images uploaded to iCloud for CSAM and reporting it to the authorities, but backpedaled after public push-back.
- Zak 2mo ago
  There were two different technologies. One was client-side scanning for known CSAM, which created a huge backlash and is described here: https://educatedguesswork.org/posts/apple-csam-intro/ https://educatedguesswork.org/posts/apple-csam-intro/ The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial and is described here: https://support.apple.com/en-us/105069 https://support.apple.com/en-us/105069
  - EmbarrassedHelp 2mo ago
    The controversial part is having the system enabled by default with age verification required to turn it off, and having the system impact non-Apple/Google apps. The UK for example wants Apple and Google to forcibly enable nudity blocking on all devices in the UK, and they want the system to bypass app/DRM security to scan all content visible on a device.
  - tzs 2mo ago
    > The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial [...] That's a new version. The one that as announced the same time as client side scanning to block uploading CSAM to iCloud worked like this. 1. It could be enabled on a child's device by the parents. It was not on be default. 2. If the child received a sexual image (not necessarily just CSAM...if an adult sends a dick pic to a child that is not CSAM but would have been flagged) the image is blocked, the child is notified, told their parents are worried the image may harm them, and asked if they still want to see it. 3. If the child says no, they do not want to see it, that is the end of the matter. 4. If the child says that they do want to see it and they are at least 13 they are shown the image and that is the end of the matter. 5. If the child says that they do want to see it and they are under 13, they are again told that they parents are concerned, and that if they view it their parents will be notified, and asked if they still want to view it. 6. If they say no that is the end of the matter. 7. If they say yes they see it but the parents also are notified and will be able to see it. This should have been pretty uncontroversial, but there were objections on the grounds that if someone say sends their dick pic to your under 13 child and the child goes all the way through to step 7 and decides to view it, that is a violation of the sender's privacy because that message was only intended for the child.
FireBeyond 2mo ago
I still also totally don't get their policy. Trying to avoid false positives by not firing until a threshold was hit (was it 20 images?) seemed insane from a PR position... rightly or wrongly, all it would take would be the wrong court case and you can see the headlines: "Apple says users can have up to 20 CSAM images on their phone before they'll tell police"
- xphos 2mo ago
  Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM? I think the program would have to say use but morally I'd say no. The issue with client scanning is it has to assume the worst, or they are than liable. If its the person has 20+ different baptism of nude babys well huh that actually might be CSAM because the context of how that concentrated photos implies but even than its hard what if that person actually has 20 God Children its less crazy than one thinks... Especially if they have multiply phones from a single baptism. You might not like pictures that way but honestly I think more important in procescuting CSAM is to go after the large sources of CSAM generation. Its trafficing in East Asia, and in Europe. I think weirdly America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures. Abuse definitely happens in the US but making policy decisions like this produces bad policy. Does iCloud rehost the photos to other people I don't really know because I use andriod tbh. If they are being rehosted (I assume to members of your contacts) that can be problematic but I think honestly the issue a lot more complex than just protect the children which the source of critic is a lot attacks against apples are coming from
  - nylonstrung 2mo ago
    > America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures The US has the largest pornography industry in the world by a massive margin, and the largest consumption of online pornography per capita Meanwhile should we be surprised that CSAM production is higher in countries like the Philippines that have very weak digital policing, abject poverty, high numbers of street children etc?
winningChild 2mo ago
[dead]
GeekyBear 2mo ago
> I feel that Apple open pandora's box with the client-side scanning. That box has been open for years now. Big brother is already watching what you do on your Android device. > A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html https://www.nytimes.com/2022/08/21/technology/google-surveil...
- trvz 2mo ago
  Google is not Apple. Apple customers expect the higher standard.
  - brokenmachine 2mo ago
    Well, there's a lot of things you can expect from Apple customers.
- letmevoteplease 2mo ago
  This is not client side. Images just sitting on your Android phone are probably safe (although Google could push an update at any time). Your images get scanned when you back them up, send them over RCS, etc. I have even seen criminal cases originating from reverse image search - anything that touche the servers of the big tech companies, except apparently Apple, will be scanned using questionable AI and against a secret list to Protect the Children.
  - GeekyBear 2mo ago
    This is an image that he did not send off of his device to anyone except his doctor's office, yet Google reached into his private data and scanned it anyway. Google reported him to the police based on a single false positive. To add insult to injury, even after the police contacted Google to tell them that they had cleared him of wrongdoing, Google refused to restore access to his account.
    - izacus 2mo ago
ribosometronome 2mo ago
The Pandora's box was already open and essentially no one noticed nor was there immense pushback that resulted in the features being removed. Photo scanning was already happening for both Android and Apple for the purpose of image search.
inigyou 2mo ago
Engineers tend to be too people-pleasing. When someone from management asks "can you scan devices for CSAM without violating privacy?" the answer should not be "hmm... well... maybe if we did it this way it could work". The answer you should give us "no" or to hedge your bets "I don't think so". Same to the government and the court. In particular a court isn't asking for a statement of objective reality whether it's possible, they're asking whether you think you could do it.
- notnullorvoid 2mo ago
  People pleasing is a component for some, some also like the technical challenge. The biggest factor though is that as a software engineer "No" is basically never an accepted answer (even from management with an engineering background). They'll either counter with "we need to do X, so find a way", or they'll move along the line to the next person until they get an answer they were looking for. Edit: There are some rare competent managers who will stop the enquiry before even asking the engineers.

by avidiax 2mo ago

mmmlinux 2mo ago

Is it different than telling your therapist something in confidence and then finding police waiting for you in the lobby.

arcticbull 2mo ago
Yes, in the sense that you have a legal doctor-patient privilege that binds what they can share with whom. There's not really an Apple cloud user privilege. No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.
- busterarm 2mo ago
  > No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc. And therapists are legally mandated to report you if you told them you viewed or possessed CSAM.
  - Dylan16807 2mo ago
    No matter how or why? That seems like a terrible mandate.
    - busterarm 2mo ago
      > No matter how or why? That seems like a terrible mandate. Honestly shocked that anyone would even say this, but even giving you the benefit of the doubt here -- the one case where I could imagine this might not happen would be if you're a police officer investigating such cases. But they also have their own therapists dedicated/trained in police-specific issues.
      - Dylan16807 2mo ago
        Even if someone went browsing for it, yes that's illegal but there's no benefit in their therapist reporting them for just visiting terrible websites. But also there are definitely ways to get accidentally exposed. That's an absolutely awful thing to call the cops over.
        busterarm 2mo ago
        I think you're demonstrating incredibly poor judgment here. CSAM is a crime with real victims. Even if your patient came across it innocently, someone is out there intentionally distributing it and that needs to be investigated.
        AnthonyMouse 2mo ago
        Have you considered the implications of what you're saying? A whistleblower goes to a therapist, stressed out over their pending decision to reveal official misconduct. They've been investigating ways to post something on the internet that can't be immediately taken down by the corrupt government officials they want to expose. They express their discomfort, in confidence, to their therapist, about using something they've discovered is also used for CSAM. You think it's a good thing for the therapist to be required to report this? Should they report that the patient admitted to viewing CSAM with no context so the whistleblower gets investigated and arrested, or should they provide the context -- that the patient is about to expose the corruption of the government receiving the report? For that matter, consider what it does when someone is actually a pedophile. They find out that if they try to seek therapy to address their perverse attraction to kids, the therapist isn't allowed to keep their confidence and they'll be arrested, so instead of seeking professional help, they keep abusing kids. Is that the result we wanted? There is a reason doctor-patient confidentiality was a thing.
  - freehorse 2mo ago
    That's not generally true. From [0] > Across most states, viewing CSEM alone is generally not a mandated-reporting trigger; reporting becomes obligatory when disclosures involve an identifiable child being abused or used to produce material. > California’s CANRA imposes a distinct duty to report electronic access (download/stream) with identifying patient information, upheld against privacy challenges based on compelling state interest. [0] https://www.psychiatrictimes.com/view/mandatory-reporting-child-sexual-exploitation-material https://www.psychiatrictimes.com/view/mandatory-reporting-ch...
- AlexandrB 2mo ago
  A better analogy is a storage locker. AFAIK police need a warrant to search "your" storage locker even though it's on someone else's property. I don't see why data in the cloud should be any different. Pre-emptively scanning everyone's data is equivalent to officers rummaging through all the storage lockers in a facility "just in case" they find something illegal.
  - d1sxeyes 2mo ago
    It’s a bit more like requiring you to submit to a weapons pat down before you go to your locker I think.

bayindirh 2mo ago

> It proved that it was technically feasible, and was "privacy preserving". Didn't their paper disproved by reversing the perceptual hashes to reveal blurred version of the images being hashed, and Apple basically said "that's fair, it's not as robust as we wanted, let's visit this later"? If not, I'll happily stand corrected, but please share sources. Addenda: - Apple's original paper: https://web.archive.org/web/20210807165030/https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://web.archive.org/web/20210807165030/https://www.apple... - Paper breaking the hash: https://arxiv.org/abs/2111.06628 https://arxiv.org/abs/2111.06628 Edit: The second one is the wrong paper. I’ll find and link the correct one tomorrow. Keeping the link for transparency.

yogorenapan 2mo ago
> to reveal blurred version of the images being hashed Skimmed your linked paper. It seems they were able to classify hashes up to ~8% top-1 accuracy and ~30% top-10. Not exactly a blurred version, or any images at all. So for example, they can say that you probably have images of trees, or images of buildings, but without much other data & very low accuracy. I'd still be a lot more concerned about them simply flagging political images rather than trying to get a broad understanding of what type of photos I have
- soulofmischief 2mo ago
  It starts at a broad understanding and ends with people permanently giving up their right and ability to keep rogue corporate governments in check.
comex 2mo ago
The paper you linked doesn’t reveal blurred versions of the images being hashed. It does train a classifier to determine which of 1,000 ImageNet classes an image belongs to, which “achieved a top-1 test accuracy of 4.34%”.
- bayindirh 2mo ago
  Then, that’s the wrong paper. I’ll find it and link it as a reply to this comment. Probably tomorrow morning.

trollbridge 2mo ago

I thought the client side scanning was to protect children? If it suspects an image is bad, it blurs it and pops up a warning including a link to resources to go to for help. Very different than trying to narc out users to the authorities.

pavon 2mo ago
That is what they actually deployed. They were planning on performing client-side scanning of all images uploaded to iCloud for CSAM and reporting it to the authorities, but backpedaled after public push-back.
Zak 2mo ago
There were two different technologies. One was client-side scanning for known CSAM, which created a huge backlash and is described here: https://educatedguesswork.org/posts/apple-csam-intro/ https://educatedguesswork.org/posts/apple-csam-intro/ The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial and is described here: https://support.apple.com/en-us/105069 https://support.apple.com/en-us/105069
- EmbarrassedHelp 2mo ago
  The controversial part is having the system enabled by default with age verification required to turn it off, and having the system impact non-Apple/Google apps. The UK for example wants Apple and Google to forcibly enable nudity blocking on all devices in the UK, and they want the system to bypass app/DRM security to scan all content visible on a device.
- tzs 2mo ago
  > The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial [...] That's a new version. The one that as announced the same time as client side scanning to block uploading CSAM to iCloud worked like this. 1. It could be enabled on a child's device by the parents. It was not on be default. 2. If the child received a sexual image (not necessarily just CSAM...if an adult sends a dick pic to a child that is not CSAM but would have been flagged) the image is blocked, the child is notified, told their parents are worried the image may harm them, and asked if they still want to see it. 3. If the child says no, they do not want to see it, that is the end of the matter. 4. If the child says that they do want to see it and they are at least 13 they are shown the image and that is the end of the matter. 5. If the child says that they do want to see it and they are under 13, they are again told that they parents are concerned, and that if they view it their parents will be notified, and asked if they still want to view it. 6. If they say no that is the end of the matter. 7. If they say yes they see it but the parents also are notified and will be able to see it. This should have been pretty uncontroversial, but there were objections on the grounds that if someone say sends their dick pic to your under 13 child and the child goes all the way through to step 7 and decides to view it, that is a violation of the sender's privacy because that message was only intended for the child.

FireBeyond 2mo ago

I still also totally don't get their policy. Trying to avoid false positives by not firing until a threshold was hit (was it 20 images?) seemed insane from a PR position... rightly or wrongly, all it would take would be the wrong court case and you can see the headlines: "Apple says users can have up to 20 CSAM images on their phone before they'll tell police"

xphos 2mo ago
Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM? I think the program would have to say use but morally I'd say no. The issue with client scanning is it has to assume the worst, or they are than liable. If its the person has 20+ different baptism of nude babys well huh that actually might be CSAM because the context of how that concentrated photos implies but even than its hard what if that person actually has 20 God Children its less crazy than one thinks... Especially if they have multiply phones from a single baptism. You might not like pictures that way but honestly I think more important in procescuting CSAM is to go after the large sources of CSAM generation. Its trafficing in East Asia, and in Europe. I think weirdly America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures. Abuse definitely happens in the US but making policy decisions like this produces bad policy. Does iCloud rehost the photos to other people I don't really know because I use andriod tbh. If they are being rehosted (I assume to members of your contacts) that can be problematic but I think honestly the issue a lot more complex than just protect the children which the source of critic is a lot attacks against apples are coming from
- nylonstrung 2mo ago
  > America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures The US has the largest pornography industry in the world by a massive margin, and the largest consumption of online pornography per capita Meanwhile should we be surprised that CSAM production is higher in countries like the Philippines that have very weak digital policing, abject poverty, high numbers of street children etc?

winningChild 2mo ago

[dead]

GeekyBear 2mo ago

> I feel that Apple open pandora's box with the client-side scanning. That box has been open for years now. Big brother is already watching what you do on your Android device. > A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html https://www.nytimes.com/2022/08/21/technology/google-surveil...

trvz 2mo ago
Google is not Apple. Apple customers expect the higher standard.
- brokenmachine 2mo ago
  Well, there's a lot of things you can expect from Apple customers.
letmevoteplease 2mo ago
This is not client side. Images just sitting on your Android phone are probably safe (although Google could push an update at any time). Your images get scanned when you back them up, send them over RCS, etc. I have even seen criminal cases originating from reverse image search - anything that touche the servers of the big tech companies, except apparently Apple, will be scanned using questionable AI and against a secret list to Protect the Children.
- GeekyBear 2mo ago
  This is an image that he did not send off of his device to anyone except his doctor's office, yet Google reached into his private data and scanned it anyway. Google reported him to the police based on a single false positive. To add insult to injury, even after the police contacted Google to tell them that they had cleared him of wrongdoing, Google refused to restore access to his account.
  - izacus 2mo ago

ribosometronome 2mo ago

The Pandora's box was already open and essentially no one noticed nor was there immense pushback that resulted in the features being removed. Photo scanning was already happening for both Android and Apple for the purpose of image search.

inigyou 2mo ago

Engineers tend to be too people-pleasing. When someone from management asks "can you scan devices for CSAM without violating privacy?" the answer should not be "hmm... well... maybe if we did it this way it could work". The answer you should give us "no" or to hedge your bets "I don't think so". Same to the government and the court. In particular a court isn't asking for a statement of objective reality whether it's possible, they're asking whether you think you could do it.

notnullorvoid 2mo ago
People pleasing is a component for some, some also like the technical challenge. The biggest factor though is that as a software engineer "No" is basically never an accepted answer (even from management with an engineering background). They'll either counter with "we need to do X, so find a way", or they'll move along the line to the next person until they get an answer they were looking for. Edit: There are some rare competent managers who will stop the enquiry before even asking the engineers.