6 ms·

Maybe.. apple had a lot of backlash using AI to detect CSAM.

by itake 2y ago

Cyph0n 2y ago
Wasn’t the backlash due to the fact that they were running detection on device against your private library?
- threeseed 2y ago
  Yes. As opposed to running it on their servers like they do now. And it was only for iCloud synced photos.
  - Zak 2y ago
    There's a huge gap between "we will scan our servers for illegal content" and "your device will scan your photos for illegal content" no matter the context. The latter makes the user's device disloyal to its owner.
    - aaomidi 2y ago
      And introduces avenues for state actors to force the scanning of other material. This was also during a time where Apple hadn’t pushed out e2ee for iCloud, so it didn’t even make sense.
      - shadowgovt 2y ago
        This ship has pretty much sailed. If you are storing your data in a large commercial vendor, assume a state actor is scanning it.
        kotaKat 2y ago
        I'm shocked at the amount of people I've seen on my local news getting arrested lately for it and it all comes from the same starting tip: "$service_provider sent a tip to NCMEC" or "uploaded a known-to-NCMEC hash", ranging from GMail, Google Drive, iCloud, and a few others. https://www.missingkids.org/cybertiplinedata https://www.missingkids.org/cybertiplinedata "In 2023, ESPs submitted 54.8 million images to the CyberTipline of which 22.4 million (41%) were unique. Of the 49.5 million videos reported by ESPs, 11.2 million (23%) were unique."
        shadowgovt 2y ago
        And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they. The slippery slope argument only applies if the slope is slippery. This is analogous to the police's use of genealogy and DNA data to narrow searches for murderers, who they then collected evidence on by other means. There's is risk there, but (at least in the US) you aren't going to find a lot of supporters of the anonymity of serial killers and child abusers. There are counter-arguments to be made. Germany is skittish about mass data collection and analysis because of their perception that it enabled the Nazi war machine to micro-target their victims. The US has no such cultural narrative.
        tzs 2y ago
        > And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they. I wouldn't be so sure. When Apple was going to introduce on-device scanning they actually proposed to do it in two places. • When you uploaded images to your iCloud account they proposed scanning them on your device first. This is the one that got by far the most attention. • The second was to scan incoming messages on phones that had parental controls set up. The way that would have worked is: 1. if it detects sexual images it would block the message, alert the child that the message contains material that the parents think might be harmful, and ask the child if they still want to see it. If the child says no that is the end of the matter. 2. if the child say they do want to see it and the child is at least 13 years old, the message is unblocked and that is the end of the matter. 3. if the child says they do want to see it and the child is under 13 they are again reminded that their parents are concerned about the message, again asked if they want to view it, and told that if they view it their parents will be told. If the child says no that is the end of the matter. 4. If the child says yes the message is unblocked and the parents are notified. This second one didn't get a lot of attention, probably because there isn't really much to object to. But I did see one objection from a fairly well known internet rights group. They objected to #4 on the grounds that the person sending the sex pictures to your under-13 year old child sent the message to the child, so it violates the sender's privacy for the parents to be notified.
    - FabHK 2y ago
      The choice was between "we will upload your pictures unencrypted and do with them as we like, including scan them for CSAM" vs. "we will upload your pictures encrypted and keep them encrypted, but will make sure beforehand on your device only that there's no known CSAM among it".
      - ImPostingOnHN 2y ago
        > we will upload your pictures unencrypted and do with them as we like Curious, I did not realize Apple sent themselves a copy of all my data, even if I have no cloud account and don't share or upload anything. Is that true?
        itake 2y ago
        Apple doesn't do this. But other service providers do (Dropbox, Google, etc). Other service providers can scan for CSAM from the cloud, but Apple cannot. So Apple might be one of the largest CSAM hosts in the world, due to this 'feature'.
        ImPostingOnHN 2y ago
        > Other service providers can scan for CSAM from the cloud I thought the topic was on-device scanning? The great-grandparent claim seemed to be that Apple had to choose between automatically uploading photos encrypted and not scanning them, vs. automatically uploading photos unencrypted and scanning them. The option for "just don't upload stuff at all, and don't scan it either" was conspicuously absent from the list of choices. Why, do other phone manufacturers do this auto-upload-and-scan without asking?
        Zak 2y ago
        I think FabHK is saying that Apple planned to offer iCloud users the choice of unencrypted storage with server-side scanning, or encrypted storage with client-side scanning. It was only meant to be for things uploaded to iCloud, but deploying such technologies for any reason creates a risk of expansion. Apple itself has other options, of course. It could offer encrypted or unencrypted storage without any kind of scanning, but has made the choice that it wants to actively check for CSAM in media stored on its servers.
    - itake 2y ago
      Apple is already categorizing content on your device. Maybe they don't report what categories you have. But I know if I search for "cat" it will show me pictures of cats on my phone.
      - mock-possum 2y ago
        Yeah it’s on by default and I’m not even sure how to turn off the visual lookup feature :/ Yet another reason why my next phone will be an android.
llm_trw 2y ago
Apple had a lot of backlash by using AI to scan every photo you ever took and sending it back to the mothership for more training.
sneak 2y ago
No, they had backlash against using AI on devices they don’t own to report said devices to police for having illegal files on them. There was no technical measure to ensure that the devices being searched were only being searched for CSAM, as the system can be used to search for any type of images chosen by Apple or the state. (Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse}.) That’s a very very different issue. I support big tech using AI models running on their own servers to detect CSAM on their own servers. I do not support big tech searching devices they do not own in violation of the wishes of the owners of those devices, simply because the police would prefer it that way. It is especially telling that iCloud Photos is not end to end encrypted (and uploads plaintext file content hashes even when optional e2ee is enabled) so Apple can and does scan 99.99%+ of the photos on everyone’s iPhones serverside already.
- threeseed 2y ago
  > they don’t own to report said devices to police for having illegal files on them They do this today. https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Technology_Summary.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... Every photo provider is required to report CSAM violations.
  - samatman 2y ago
    Actually they do not. https://forums.appleinsider.com/discussion/238553/apple-sued-over-2022-dropping-of-csam-detection-features https://forums.appleinsider.com/discussion/238553/apple-sued...
- skissane 2y ago
  > Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse} It hasn’t been redefined. The legal definition of it in the UK, Canada, Australia, New Zealand has included computer generated imagery since at least the 1990s. The US Congress did the same thing in 1996, but the US Supreme Court ruled in the 2002 case of Ashcroft v Free Speech Coalition that it violated the First Amendment. [0] This predates GenAI because even in the 1990s people saw where CGI was going and could foresee this kind of thing would one day be possible. Added to that: a lot of people misunderstand what that 2002 case held. SCOTUS case law establishes two distinct exceptions to the First Amendment – child pornography and obscenity. The first is easier to prosecute and more commonly prosecuted; the 2002 case held that "virtual child pornography" (made without the use of any actual children) does not fall into the scope of the child pornography exception – but it still falls into the scope of the obscenity exception. There is in fact a distinct federal crime for obscenity involving children as opposed to adults, 18 USC 1466A ("Obscene visual representations of the sexual abuse of children") [1] enacted in 2003 in response to this decision. Child obscenity is less commonly prosecuted, but in 2021 a Texas man was sentenced to 40 years in prison over it [2] – that wasn't for GenAI, that was for drawings and text, but if drawings fall into the legal category, obviously GenAI images will too. So actually it turns out that even in the US, GenAI materials can legally count as CSAM, if we define CSAM to include both child pornography and child obscenity – and this has been true since at least 2003, long before the GenAI era. [0] https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalition https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalit... [1] https://www.law.cornell.edu/uscode/text/18/1466A https://www.law.cornell.edu/uscode/text/18/1466A [2] https://www.justice.gov/opa/pr/texas-man-sentenced-40-years-prison-running-child-obscenity-website https://www.justice.gov/opa/pr/texas-man-sentenced-40-years-...
PittleyDunkin 2y ago
I don't think the problem there is the AI aspect
- itake 2y ago
  My understanding was the FP risk. Everything was on device. People designed images that were FPs of real images.
  - PittleyDunkin 2y ago
    FP? Let us know what this means when you have a chance. Federal Prosecution? Fake Porn? Fictional Pictures?
    - thinkmassive 2y ago
      My guess is False Positive. Weird abbreviation to use though.
pluc 2y ago
Probably because you need to feed it child porn so it can detect it...
- hirvi74 2y ago
  Already happened/happening. I have an ex-coworker that left my current employer for my state's version of the FBI. Long story short, the government has a massive database to crosscheck against. Often times, the would use automated processes to filter through suspicious data they would collect during arrests. If the automated process flags something as a potential hit, then they, the humans, would then review those images to verify. Every image/video that is discovered to be a hit is also inserted into a larger dataset as well. I can't remember if the Feds have their own DB (why wouldn't they?), but the National Center for Missing and Exploited Children run a database that I believe government agencies use too. Not to mention, companies like Dropbox, Google, etc.. all has against the database(s) as well.

Maybe.. apple had a lot of backlash using AI to detect CSAM.

by itake 2y ago

Maybe.. apple had a lot of backlash using AI to detect CSAM.

Cyph0n 2y ago

Wasn’t the backlash due to the fact that they were running detection on device against your private library?

threeseed 2y ago
Yes. As opposed to running it on their servers like they do now. And it was only for iCloud synced photos.
- Zak 2y ago
  There's a huge gap between "we will scan our servers for illegal content" and "your device will scan your photos for illegal content" no matter the context. The latter makes the user's device disloyal to its owner.
  - aaomidi 2y ago
    And introduces avenues for state actors to force the scanning of other material. This was also during a time where Apple hadn’t pushed out e2ee for iCloud, so it didn’t even make sense.
    - shadowgovt 2y ago
      This ship has pretty much sailed. If you are storing your data in a large commercial vendor, assume a state actor is scanning it.
      - kotaKat 2y ago
        I'm shocked at the amount of people I've seen on my local news getting arrested lately for it and it all comes from the same starting tip: "$service_provider sent a tip to NCMEC" or "uploaded a known-to-NCMEC hash", ranging from GMail, Google Drive, iCloud, and a few others. https://www.missingkids.org/cybertiplinedata https://www.missingkids.org/cybertiplinedata "In 2023, ESPs submitted 54.8 million images to the CyberTipline of which 22.4 million (41%) were unique. Of the 49.5 million videos reported by ESPs, 11.2 million (23%) were unique."
        shadowgovt 2y ago
        And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they. The slippery slope argument only applies if the slope is slippery. This is analogous to the police's use of genealogy and DNA data to narrow searches for murderers, who they then collected evidence on by other means. There's is risk there, but (at least in the US) you aren't going to find a lot of supporters of the anonymity of serial killers and child abusers. There are counter-arguments to be made. Germany is skittish about mass data collection and analysis because of their perception that it enabled the Nazi war machine to micro-target their victims. The US has no such cultural narrative.
        tzs 2y ago
        > And, indeed, this is why we should not expect the process to stop. Nobody is rallying behind the rights of child abusers and those who traffic in child abuse material. Arguably, nor should they. I wouldn't be so sure. When Apple was going to introduce on-device scanning they actually proposed to do it in two places. • When you uploaded images to your iCloud account they proposed scanning them on your device first. This is the one that got by far the most attention. • The second was to scan incoming messages on phones that had parental controls set up. The way that would have worked is: 1. if it detects sexual images it would block the message, alert the child that the message contains material that the parents think might be harmful, and ask the child if they still want to see it. If the child says no that is the end of the matter. 2. if the child say they do want to see it and the child is at least 13 years old, the message is unblocked and that is the end of the matter. 3. if the child says they do want to see it and the child is under 13 they are again reminded that their parents are concerned about the message, again asked if they want to view it, and told that if they view it their parents will be told. If the child says no that is the end of the matter. 4. If the child says yes the message is unblocked and the parents are notified. This second one didn't get a lot of attention, probably because there isn't really much to object to. But I did see one objection from a fairly well known internet rights group. They objected to #4 on the grounds that the person sending the sex pictures to your under-13 year old child sent the message to the child, so it violates the sender's privacy for the parents to be notified.
  - FabHK 2y ago
    The choice was between "we will upload your pictures unencrypted and do with them as we like, including scan them for CSAM" vs. "we will upload your pictures encrypted and keep them encrypted, but will make sure beforehand on your device only that there's no known CSAM among it".
    - ImPostingOnHN 2y ago
      > we will upload your pictures unencrypted and do with them as we like Curious, I did not realize Apple sent themselves a copy of all my data, even if I have no cloud account and don't share or upload anything. Is that true?
      - itake 2y ago
        Apple doesn't do this. But other service providers do (Dropbox, Google, etc). Other service providers can scan for CSAM from the cloud, but Apple cannot. So Apple might be one of the largest CSAM hosts in the world, due to this 'feature'.
        ImPostingOnHN 2y ago
        > Other service providers can scan for CSAM from the cloud I thought the topic was on-device scanning? The great-grandparent claim seemed to be that Apple had to choose between automatically uploading photos encrypted and not scanning them, vs. automatically uploading photos unencrypted and scanning them. The option for "just don't upload stuff at all, and don't scan it either" was conspicuously absent from the list of choices. Why, do other phone manufacturers do this auto-upload-and-scan without asking?
        Zak 2y ago
        I think FabHK is saying that Apple planned to offer iCloud users the choice of unencrypted storage with server-side scanning, or encrypted storage with client-side scanning. It was only meant to be for things uploaded to iCloud, but deploying such technologies for any reason creates a risk of expansion. Apple itself has other options, of course. It could offer encrypted or unencrypted storage without any kind of scanning, but has made the choice that it wants to actively check for CSAM in media stored on its servers.
  - itake 2y ago
    Apple is already categorizing content on your device. Maybe they don't report what categories you have. But I know if I search for "cat" it will show me pictures of cats on my phone.
    - mock-possum 2y ago
      Yeah it’s on by default and I’m not even sure how to turn off the visual lookup feature :/ Yet another reason why my next phone will be an android.

llm_trw 2y ago

Apple had a lot of backlash by using AI to scan every photo you ever took and sending it back to the mothership for more training.

sneak 2y ago

No, they had backlash against using AI on devices they don’t own to report said devices to police for having illegal files on them. There was no technical measure to ensure that the devices being searched were only being searched for CSAM, as the system can be used to search for any type of images chosen by Apple or the state. (Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse}.) That’s a very very different issue. I support big tech using AI models running on their own servers to detect CSAM on their own servers. I do not support big tech searching devices they do not own in violation of the wishes of the owners of those devices, simply because the police would prefer it that way. It is especially telling that iCloud Photos is not end to end encrypted (and uploads plaintext file content hashes even when optional e2ee is enabled) so Apple can and does scan 99.99%+ of the photos on everyone’s iPhones serverside already.

threeseed 2y ago
> they don’t own to report said devices to police for having illegal files on them They do this today. https://www.apple.com/child-safety/pdf/Expanded_Protections_for_Children_Technology_Summary.pdf https://www.apple.com/child-safety/pdf/Expanded_Protections_... Every photo provider is required to report CSAM violations.
- samatman 2y ago
  Actually they do not. https://forums.appleinsider.com/discussion/238553/apple-sued-over-2022-dropping-of-csam-detection-features https://forums.appleinsider.com/discussion/238553/apple-sued...
skissane 2y ago
> Also, with the advent of GenAI, CSAM has been redefined to include generated imagery that does not contain any of {children, sex, abuse} It hasn’t been redefined. The legal definition of it in the UK, Canada, Australia, New Zealand has included computer generated imagery since at least the 1990s. The US Congress did the same thing in 1996, but the US Supreme Court ruled in the 2002 case of Ashcroft v Free Speech Coalition that it violated the First Amendment. [0] This predates GenAI because even in the 1990s people saw where CGI was going and could foresee this kind of thing would one day be possible. Added to that: a lot of people misunderstand what that 2002 case held. SCOTUS case law establishes two distinct exceptions to the First Amendment – child pornography and obscenity. The first is easier to prosecute and more commonly prosecuted; the 2002 case held that "virtual child pornography" (made without the use of any actual children) does not fall into the scope of the child pornography exception – but it still falls into the scope of the obscenity exception. There is in fact a distinct federal crime for obscenity involving children as opposed to adults, 18 USC 1466A ("Obscene visual representations of the sexual abuse of children") [1] enacted in 2003 in response to this decision. Child obscenity is less commonly prosecuted, but in 2021 a Texas man was sentenced to 40 years in prison over it [2] – that wasn't for GenAI, that was for drawings and text, but if drawings fall into the legal category, obviously GenAI images will too. So actually it turns out that even in the US, GenAI materials can legally count as CSAM, if we define CSAM to include both child pornography and child obscenity – and this has been true since at least 2003, long before the GenAI era. [0] https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalition https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalit... [1] https://www.law.cornell.edu/uscode/text/18/1466A https://www.law.cornell.edu/uscode/text/18/1466A [2] https://www.justice.gov/opa/pr/texas-man-sentenced-40-years-prison-running-child-obscenity-website https://www.justice.gov/opa/pr/texas-man-sentenced-40-years-...

PittleyDunkin 2y ago

I don't think the problem there is the AI aspect

itake 2y ago
My understanding was the FP risk. Everything was on device. People designed images that were FPs of real images.
- PittleyDunkin 2y ago
  FP? Let us know what this means when you have a chance. Federal Prosecution? Fake Porn? Fictional Pictures?
  - thinkmassive 2y ago
    My guess is False Positive. Weird abbreviation to use though.

pluc 2y ago

Probably because you need to feed it child porn so it can detect it...

hirvi74 2y ago
Already happened/happening. I have an ex-coworker that left my current employer for my state's version of the FBI. Long story short, the government has a massive database to crosscheck against. Often times, the would use automated processes to filter through suspicious data they would collect during arrests. If the automated process flags something as a potential hit, then they, the humans, would then review those images to verify. Every image/video that is discovered to be a hit is also inserted into a larger dataset as well. I can't remember if the Feds have their own DB (why wouldn't they?), but the National Center for Missing and Exploited Children run a database that I believe government agencies use too. Not to mention, companies like Dropbox, Google, etc.. all has against the database(s) as well.