5 ms·

I think, but mostly hope, that this client side scanning initiative will fall apart. Not just because it's an obvious destruction of civil liberties for us in t

I think, but mostly hope, that this client side scanning initiative will fall apart. Not just because it's an obvious destruction of civil liberties for us in the EU, but also because it's doomed to fail even if implemented. The underlying concept is ineffective at dealing with CSAM, or any kind of adversarial user scenario. Unfortunately my hopes have a bad legislative record. Clients aren't trustworthy. Ignoring the myriad services that exist, and will come to exist, which just never implement this -- there will be (somewhat) easily accessible versions of apps that remove such telemetry. E.g. VScode is to VScodium as WhatsApp is to WhatsAppium. Imagine the spoofing opportunities. Message someone, get their client ID, report that client detected all the worst CSAM imaginable. Think of the false positives. Think of the expense. Think of the children. Think of the children whose non-abusive parents send banal but confounding photos to each other, get hounded by CSAM allegations, and some fraction of whom get separated from their child. Passing this will be a huge expense of time all of these legislators could use better. Implementing these directives will be a huge expense of money and knowhow to build a flawed and exploitable system. Enforcing the consequences will have untold amounts of suffering unrelated to CSAM. And likely after the first few creeps are caught, the adversarial actors will just start using other channels. But by that point the conversation will have shifted to clientside scanning of thoughts...

ChrisKnott 3y ago
> "Think of the children whose non-abusive parents send banal but confounding photos to each other, get hounded by CSAM allegations, and some fraction of whom get separated from their child." How do you see a false positive progressing into child separation? People who are actually convicted of possessing CSAM don't necessarily lose access to their children at the moment.
- matthewdgreen 3y ago
  Ever since I started writing on this issue, I’ve had a number of folks contact me and say they were unwittingly flagged by Google Photos’ ML-based CSAM detection algorithm. The most recent of these told me the photo was sent as a (pretty awful) prank in a WhatsApp group, and their phone auto-saved it to the camera roll. All of these folks lost all access to Google services with minimal options for appeal, and have had their lives disrupted in various ways I can’t imagine. I have no way of verifying what these people are telling me, and I can’t help them even if I did. And even more than this: my first reaction is to wonder if they’re all actually pedophiles who are just lying to me, and/or if they’re going to text CSAM at me. (Because that’s the power of this accusation — nobody will trust you once they hear it.) Some of these folks have had their lives ruined and some have told me they were hospitalized and considered suicide, even though they weren’t actually prosecuted. I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) If none of this stuff convinces you, here is a much more carefully-documented version of a similar story in which a dad took a photo of their toddler’s rash and was subsequently investigated for CSAM. This one has the “happy” ending that the accused is entirely cleared by the police, but never gets his Google account back and has to live with whatever other trauma and shame comes from the investigation: https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?smid=nytcore-ios-share&referringSource=articleShare https://www.nytimes.com/2022/08/21/technology/google-surveil...
  - ChrisKnott 3y ago
    I understand how the system can lead to people losing access to their Google account, GP suggested people could lose access to their children. The NYT article is the opposite of what GP claimed - the case was closed before he even knew it had been opened.
    - matthewdgreen 3y ago
      Imagine you are in the middle of a custody battle when this news comes down, or that you are jailed before being cleared. Imagine the cops and social workers in your city aren't diligent, or that you already have a criminal record. And further imagine you don’t have the luxury of a New York Times reporter investigating your case and writing publicly about how you’re innocent. The NYT example is a best case outcome for an innocent person who triggers these algorithms.
      - ChrisKnott 3y ago
        I guess my point is I don't see how your "jailed before being cleared" example could ever reasonably happen. The NYT is a best case, but also an average case, and in fact any other outcome would be absurd.
        yukkuri 3y ago
        The system isn't reasonable, so your denial based on what you think can "reasonably" happen is just your privilege talking.
        ChrisKnott 3y ago
        That might be the case, but there would still be examples of it happening to other people, wouldn't there?
        tremon 3y ago
        Re: unreasonable system, would you accept as evidence the 1115 children that were removed from their parents because the Dutch IRS labeled the parents as fraudsters (due to a biased and discriminatory AI system, of course)? https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-van-het-toeslagenschandaal-uit-huis-geplaatst-worden https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-va...
- Humorist2290 3y ago
  In practice, I imagine it would be in the minority of cases when law enforcement is given a report on someone including this detection "evidence," and for the duration of the investigation the child is placed in protective custody. Not necessarily permanently, however the experience of family separation even for a few days can be traumatic. Criminal investigations can often take more than a few days of course. At the same time, there are unfortunately many counterexamples of abusive parents not being separated from their children via the same mechanisms. Sometimes this is from the abusers being "good criminals" and not getting caught; but sometimes it's from LEOs being bad LEOs and not doing the catching. My point was that in a system which surveils ~500M people, even an FP rate of ~0.01% is 50k people. Supposing the LEOs tasked with filtering this into credible (and triable) cases further manage to, through amazing police work, confirm the innocence of 99% of those false positives. That's still 500 falsely accused, but now confirmed suspicious, suspects.
  - ChrisKnott 3y ago
    I had assumed the report sent to law enforcement would include the actual image sent (the suspected CSAM). This is what happens at the moment with NCMEC. The comments from Alexander Hanff seem to suggest this as well.
- XorNot 3y ago
  Indiscriminate surveillance combined with an intent assumption: the system is setup to find CSAM, so it doesn't test alternate hypotheses. There's many intervening steps[1] of consequences which can result. Not to mention chilling effects: it's a standard bit of knowledge now that all my family members worry a bit if they take a photo of my son doing something and he's not wearing a shirt or something. Because no one's entirely sure the extent or impact of whatever scanning systems already exist (hey folks: encrypt everything on the cloud, and thank god for Signal). [1] https://au.pcmag.com/security/95768/nyt-parents-lose-google-accounts-over-abuse-image-false-positives https://au.pcmag.com/security/95768/nyt-parents-lose-google-...

ChrisKnott 3y ago

> "Think of the children whose non-abusive parents send banal but confounding photos to each other, get hounded by CSAM allegations, and some fraction of whom get separated from their child." How do you see a false positive progressing into child separation? People who are actually convicted of possessing CSAM don't necessarily lose access to their children at the moment.

matthewdgreen 3y ago
Ever since I started writing on this issue, I’ve had a number of folks contact me and say they were unwittingly flagged by Google Photos’ ML-based CSAM detection algorithm. The most recent of these told me the photo was sent as a (pretty awful) prank in a WhatsApp group, and their phone auto-saved it to the camera roll. All of these folks lost all access to Google services with minimal options for appeal, and have had their lives disrupted in various ways I can’t imagine. I have no way of verifying what these people are telling me, and I can’t help them even if I did. And even more than this: my first reaction is to wonder if they’re all actually pedophiles who are just lying to me, and/or if they’re going to text CSAM at me. (Because that’s the power of this accusation — nobody will trust you once they hear it.) Some of these folks have had their lives ruined and some have told me they were hospitalized and considered suicide, even though they weren’t actually prosecuted. I can certainly imagine being in their situation due to the mistake of having an asshole friend-of-a-friend in a group chat and some bad settings in my phone. (I have neither, thankfully.) If none of this stuff convinces you, here is a much more carefully-documented version of a similar story in which a dad took a photo of their toddler’s rash and was subsequently investigated for CSAM. This one has the “happy” ending that the accused is entirely cleared by the police, but never gets his Google account back and has to live with whatever other trauma and shame comes from the investigation: https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?smid=nytcore-ios-share&referringSource=articleShare https://www.nytimes.com/2022/08/21/technology/google-surveil...
- ChrisKnott 3y ago
  I understand how the system can lead to people losing access to their Google account, GP suggested people could lose access to their children. The NYT article is the opposite of what GP claimed - the case was closed before he even knew it had been opened.
  - matthewdgreen 3y ago
    Imagine you are in the middle of a custody battle when this news comes down, or that you are jailed before being cleared. Imagine the cops and social workers in your city aren't diligent, or that you already have a criminal record. And further imagine you don’t have the luxury of a New York Times reporter investigating your case and writing publicly about how you’re innocent. The NYT example is a best case outcome for an innocent person who triggers these algorithms.
    - ChrisKnott 3y ago
      I guess my point is I don't see how your "jailed before being cleared" example could ever reasonably happen. The NYT is a best case, but also an average case, and in fact any other outcome would be absurd.
      - yukkuri 3y ago
        The system isn't reasonable, so your denial based on what you think can "reasonably" happen is just your privilege talking.
        ChrisKnott 3y ago
        That might be the case, but there would still be examples of it happening to other people, wouldn't there?
        tremon 3y ago
        Re: unreasonable system, would you accept as evidence the 1115 children that were removed from their parents because the Dutch IRS labeled the parents as fraudsters (due to a biased and discriminatory AI system, of course)? https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-van-het-toeslagenschandaal-uit-huis-geplaatst-worden https://www.bnnvara.nl/artikelen/hoe-konden-1115-kinderen-va...
Humorist2290 3y ago
In practice, I imagine it would be in the minority of cases when law enforcement is given a report on someone including this detection "evidence," and for the duration of the investigation the child is placed in protective custody. Not necessarily permanently, however the experience of family separation even for a few days can be traumatic. Criminal investigations can often take more than a few days of course. At the same time, there are unfortunately many counterexamples of abusive parents not being separated from their children via the same mechanisms. Sometimes this is from the abusers being "good criminals" and not getting caught; but sometimes it's from LEOs being bad LEOs and not doing the catching. My point was that in a system which surveils ~500M people, even an FP rate of ~0.01% is 50k people. Supposing the LEOs tasked with filtering this into credible (and triable) cases further manage to, through amazing police work, confirm the innocence of 99% of those false positives. That's still 500 falsely accused, but now confirmed suspicious, suspects.
- ChrisKnott 3y ago
  I had assumed the report sent to law enforcement would include the actual image sent (the suspected CSAM). This is what happens at the moment with NCMEC. The comments from Alexander Hanff seem to suggest this as well.
XorNot 3y ago
Indiscriminate surveillance combined with an intent assumption: the system is setup to find CSAM, so it doesn't test alternate hypotheses. There's many intervening steps[1] of consequences which can result. Not to mention chilling effects: it's a standard bit of knowledge now that all my family members worry a bit if they take a photo of my son doing something and he's not wearing a shirt or something. Because no one's entirely sure the extent or impact of whatever scanning systems already exist (hey folks: encrypt everything on the cloud, and thank god for Signal). [1] https://au.pcmag.com/security/95768/nyt-parents-lose-google-accounts-over-abuse-image-false-positives https://au.pcmag.com/security/95768/nyt-parents-lose-google-...