6 ms·

> "Think of the children whose non-abusive parents send banal but confounding photos to each other, get hounded by CSAM allegations, and some fraction of whom g

> "Think of the children whose non-abusive parents send banal but confounding photos to each other, get hounded by CSAM allegations, and some fraction of whom get separated from their child." How do you see a false positive progressing into child separation? People who are actually convicted of possessing CSAM don't necessarily lose access to their children at the moment.

matthewdgreen 3y ago
Ever since I started writing on this issue, I’ve had a number of folks contact me and say they were unwittingly flagged by Google Photos’ ML-based CSAM detection algorithm. The most recent of these told me the photo was sent as a (pretty awful) prank in a WhatsApp group, and their phone auto-saved it to the camera roll. All of these folks lost all access to Google services with minimal options for appeal, and have had their lives disrupted in various ways I can’t imagine. I have no way of verifying what these people are telling me, and I can’t help them even if I did. And even more than this: my first reaction is to wonder if they’re all actually pedophiles who are just lying to me, and/or if they’re going to text CSAM at me. (Because that’s the power of this accusation — nobody will trust you once they hear it.) Some of these folks have had their lives ruined and some have told me they were hospitalized and considered suicide, even though they weren’t actually prosecuted. I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) If none of this stuff convinces you, here is a much more carefully-documented version of a similar story in which a dad took a photo of their toddler’s rash and was subsequently investigated for CSAM. This one has the “happy” ending that the accused is entirely cleared by the police, but never gets his Google account back and has to live with whatever other trauma and shame comes from the investigation: https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?smid=nytcore-ios-share&referringSource=articleShare https://www.nytimes.com/2022/08/21/technology/google-surveil...
- ChrisKnott 3y ago
  I understand how the system can lead to people losing access to their Google account, GP suggested people could lose access to their children. The NYT article is the opposite of what GP claimed - the case was closed before he even knew it had been opened.
  - matthewdgreen 3y ago
    Imagine you are in the middle of a custody battle when this news comes down, or that you are jailed before being cleared. Imagine the cops and social workers in your city aren't diligent, or that you already have a criminal record. And further imagine you don’t have the luxury of a New York Times reporter investigating your case and writing publicly about how you’re innocent. The NYT example is a best case outcome for an innocent person who triggers these algorithms.
    - ChrisKnott 3y ago
      I guess my point is I don't see how your "jailed before being cleared" example could ever reasonably happen. The NYT is a best case, but also an average case, and in fact any other outcome would be absurd.
      - yukkuri 3y ago
        The system isn't reasonable, so your denial based on what you think can "reasonably" happen is just your privilege talking.
        ChrisKnott 3y ago
        That might be the case, but there would still be examples of it happening to other people, wouldn't there?
        tremon 3y ago
        Re: unreasonable system, would you accept as evidence the 1115 children that were removed from their parents because the Dutch IRS labeled the parents as fraudsters (due to a biased and discriminatory AI system, of course)? https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-van-het-toeslagenschandaal-uit-huis-geplaatst-worden https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-va...
- wepple 3y ago
  That’s an utter nightmare of a situation I’d never considered, or prepared to defend against. I hope the “prankster” ends up caught and in prison
- omginternets 3y ago
  It’s even worse than this because once the idea of mass CSAM-scanning is normalized, a false accusation is immediately credible (they would know, after all…)
- acdha 3y ago
  > I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) This always makes me think how much worse middle school will get: kids are guaranteed to push boundaries, they all have some anger justified or not towards each other or the adults they know, and some fraction will learn how nastily they can weaponize the mandatory reporter system. Even if the truth comes out quickly, there’s so much room for strife first.
Humorist2290 3y ago
In practice, I imagine it would be in the minority of cases when law enforcement is given a report on someone including this detection "evidence," and for the duration of the investigation the child is placed in protective custody. Not necessarily permanently, however the experience of family separation even for a few days can be traumatic. Criminal investigations can often take more than a few days of course. At the same time, there are unfortunately many counterexamples of abusive parents not being separated from their children via the same mechanisms. Sometimes this is from the abusers being "good criminals" and not getting caught; but sometimes it's from LEOs being bad LEOs and not doing the catching. My point was that in a system which surveils ~500M people, even an FP rate of ~0.01% is 50k people. Supposing the LEOs tasked with filtering this into credible (and triable) cases further manage to, through amazing police work, confirm the innocence of 99% of those false positives. That's still 500 falsely accused, but now confirmed suspicious, suspects.
- ChrisKnott 3y ago
  I had assumed the report sent to law enforcement would include the actual image sent (the suspected CSAM). This is what happens at the moment with NCMEC. The comments from Alexander Hanff seem to suggest this as well.
XorNot 3y ago
Indiscriminate surveillance combined with an intent assumption: the system is setup to find CSAM, so it doesn't test alternate hypotheses. There's many intervening steps[1] of consequences which can result. Not to mention chilling effects: it's a standard bit of knowledge now that all my family members worry a bit if they take a photo of my son doing something and he's not wearing a shirt or something. Because no one's entirely sure the extent or impact of whatever scanning systems already exist (hey folks: encrypt everything on the cloud, and thank god for Signal). [1] https://au.pcmag.com/security/95768/nyt-parents-lose-google-accounts-over-abuse-image-false-positives https://au.pcmag.com/security/95768/nyt-parents-lose-google-...

matthewdgreen 3y ago

Ever since I started writing on this issue, I’ve had a number of folks contact me and say they were unwittingly flagged by Google Photos’ ML-based CSAM detection algorithm. The most recent of these told me the photo was sent as a (pretty awful) prank in a WhatsApp group, and their phone auto-saved it to the camera roll. All of these folks lost all access to Google services with minimal options for appeal, and have had their lives disrupted in various ways I can’t imagine. I have no way of verifying what these people are telling me, and I can’t help them even if I did. And even more than this: my first reaction is to wonder if they’re all actually pedophiles who are just lying to me, and/or if they’re going to text CSAM at me. (Because that’s the power of this accusation — nobody will trust you once they hear it.) Some of these folks have had their lives ruined and some have told me they were hospitalized and considered suicide, even though they weren’t actually prosecuted. I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) If none of this stuff convinces you, here is a much more carefully-documented version of a similar story in which a dad took a photo of their toddler’s rash and was subsequently investigated for CSAM. This one has the “happy” ending that the accused is entirely cleared by the police, but never gets his Google account back and has to live with whatever other trauma and shame comes from the investigation: https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?smid=nytcore-ios-share&referringSource=articleShare https://www.nytimes.com/2022/08/21/technology/google-surveil...

ChrisKnott 3y ago
I understand how the system can lead to people losing access to their Google account, GP suggested people could lose access to their children. The NYT article is the opposite of what GP claimed - the case was closed before he even knew it had been opened.
- matthewdgreen 3y ago
  Imagine you are in the middle of a custody battle when this news comes down, or that you are jailed before being cleared. Imagine the cops and social workers in your city aren't diligent, or that you already have a criminal record. And further imagine you don’t have the luxury of a New York Times reporter investigating your case and writing publicly about how you’re innocent. The NYT example is a best case outcome for an innocent person who triggers these algorithms.
  - ChrisKnott 3y ago
    I guess my point is I don't see how your "jailed before being cleared" example could ever reasonably happen. The NYT is a best case, but also an average case, and in fact any other outcome would be absurd.
    - yukkuri 3y ago
      The system isn't reasonable, so your denial based on what you think can "reasonably" happen is just your privilege talking.
      - ChrisKnott 3y ago
        That might be the case, but there would still be examples of it happening to other people, wouldn't there?
        tremon 3y ago
        Re: unreasonable system, would you accept as evidence the 1115 children that were removed from their parents because the Dutch IRS labeled the parents as fraudsters (due to a biased and discriminatory AI system, of course)? https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-van-het-toeslagenschandaal-uit-huis-geplaatst-worden https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-va...
wepple 3y ago
That’s an utter nightmare of a situation I’d never considered, or prepared to defend against. I hope the “prankster” ends up caught and in prison
omginternets 3y ago
It’s even worse than this because once the idea of mass CSAM-scanning is normalized, a false accusation is immediately credible (they would know, after all…)
acdha 3y ago
> I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) This always makes me think how much worse middle school will get: kids are guaranteed to push boundaries, they all have some anger justified or not towards each other or the adults they know, and some fraction will learn how nastily they can weaponize the mandatory reporter system. Even if the truth comes out quickly, there’s so much room for strife first.

Humorist2290 3y ago

In practice, I imagine it would be in the minority of cases when law enforcement is given a report on someone including this detection "evidence," and for the duration of the investigation the child is placed in protective custody. Not necessarily permanently, however the experience of family separation even for a few days can be traumatic. Criminal investigations can often take more than a few days of course. At the same time, there are unfortunately many counterexamples of abusive parents not being separated from their children via the same mechanisms. Sometimes this is from the abusers being "good criminals" and not getting caught; but sometimes it's from LEOs being bad LEOs and not doing the catching. My point was that in a system which surveils ~500M people, even an FP rate of ~0.01% is 50k people. Supposing the LEOs tasked with filtering this into credible (and triable) cases further manage to, through amazing police work, confirm the innocence of 99% of those false positives. That's still 500 falsely accused, but now confirmed suspicious, suspects.

ChrisKnott 3y ago
I had assumed the report sent to law enforcement would include the actual image sent (the suspected CSAM). This is what happens at the moment with NCMEC. The comments from Alexander Hanff seem to suggest this as well.

XorNot 3y ago

Indiscriminate surveillance combined with an intent assumption: the system is setup to find CSAM, so it doesn't test alternate hypotheses. There's many intervening steps[1] of consequences which can result. Not to mention chilling effects: it's a standard bit of knowledge now that all my family members worry a bit if they take a photo of my son doing something and he's not wearing a shirt or something. Because no one's entirely sure the extent or impact of whatever scanning systems already exist (hey folks: encrypt everything on the cloud, and thank god for Signal). [1] https://au.pcmag.com/security/95768/nyt-parents-lose-google-accounts-over-abuse-image-false-positives https://au.pcmag.com/security/95768/nyt-parents-lose-google-...