5 ms·

What would folks think of this compromise? Do the AI threat detection stuff on the device, and warn the user if suspected abusive images/text are detected?

by hgsgm 4y ago

olliej 4y ago
The governments don't want the user to be told, they want the government to be told. There are already numerous stories of people suffering tremendous harm because of the systems that are already in place for this purpose. Do you have a photo of your tiny kids playing in a pool? better hope google doesn't report you to the police and permanently ban you from all their services and delete all your data.
- singleshot_ 4y ago
  I think it’s far, far worse than this. Have you ever been inconvenienced because of a software bug? Now imagine you’re in jail for child sexual exploitation. Have you ever tried to contact one of these big service providers to get something fixed? Would you like Google tech support (snort) be the last thing standing between you and a prison sentence? I think the absolute last place we want to be is in a world where we can’t make money to eat without our portable computer/phone that runs broken software made by some scammy dot com whose salient feature is that if we don’t stay ahead of it, it will imprison the user.
  - olliej 4y ago
    I agree entirely, but I used the google example because they've specifically done exactly that: * Reported an image for being CSAM * Terminated the account * The police correctly recognized it isn't CSAM * Google refused to reinstate the account But even if you assume a functioning and rule-of-law based legal system and you don't go to jail without actually committing a crime, the cost of bogus CSAM reports is in no way lost on me, especially in the conservative US and many similar regressive regimes around the world where LGBT anything is legislated as child abuse. Even if you are found innocent (which again, regressive regimes make hard) and news reports and anything on the internet will be "X was arrested for having child abuse imagery" nothing will ever be updated to reflect the result. This is ignoring what happens if people maliciously inject child abuse into your device's "CSAM" filter. Before the "CSAM" detection gets expanded to "weapons", "graffiti", "evidence of crime", etc. Take the UK where politicians aggressively passed laws to allow spying to prevent "terrorism", and those laws are used almost exclusively for councils to enforce local ordinances related to cleaning, taxes, and fees. And of course none of those laws were needed to prevent the crimes they're ostensibly meant to stop (literally multiple reports to the police prior to Manchester, etc).
- throwaway67743 4y ago
  There is a solution to the latter point, don't use malicious companies like Google, because really they're awful in so many other ways this is way down on the list.
  - olliej 4y ago
    Who would you propose instead of google, given they'd all [in this scenario] be subject to the same laws regarding scanning and notification?
    - throwaway67743 4y ago
      You can host whatever it is yourself which exempts you from most obligations, and even if not you're free to ignore the rules because nobody cares about you as an individual. There are enough alternatives also these days that may not be obligated due to size or jurisdiction etc (eg incoming EU regulations require a minimum user count, and such) The USA is unlikely to blanket ban encryption due to the way the country is organised from a federal and state point of view, and others like the UK are basically irrelevant as nobody will rule in their favour over domestic entities, as much as they'd like to believe otherwise and many countries just do not care about anyone else so that is always an option.
      - olliej 4y ago
        The whole point of on device detection is that no hosting is involved or required: the media is scanned on device, and if the media matches the criminal material (actual child abuse, but in many US states easily anything transition or LGBT related, etc in other states anything gun or weapon related, etc) that is directly reported to law enforcement. Your ability to use an EU service gets exceedingly restricted when the US requires all EU businesses that serve US customers to comply with US law.
        throwaway67743 4y ago
        Well sure but the parent comment was re Google, who do cloud scanning already, that can be avoided by doing it yourself - device scanning is pretty pointless as the people who it's "intended" to catch won't use devices that have that capability and more likely it'll just be used by the more authoritarian states to curb dissent etc, or put people in jail for having a meme with Hitler in it... The USA is still probably safe there, the EU do what they want anyway without any regard for the impact it has. (From the POV of an EU citizen with US legal entities, so I'm getting shafted from both sides)
tadfisher 4y ago
How about the users receiving abusive images/text report them to the government when received? Don't need to break E2EE for that.
- Paul-Craft 4y ago
  You're assuming the recipients don't want to receive said data.
singleshot_ 4y ago
I would think that’s a profoundly intrusive way to treat 330 million citizens of a liberal democracy against whom you have discovered no reasonable suspicion and developed no probable cause. We have a fourth amendment for a reason and it’s to prevent evil tyrants from oppressing us. Run whatever you want on your phone but this is a hard pass from me.
waselighis 4y ago
You may already be aware, but Apple tried to implement this on their devices a couple years ago. "Client side scanning". There was huge public backlash and Apple eventually moved away from the idea.

What would folks think of this compromise? Do the AI threat detection stuff on the device, and warn the user if suspected abusive images/text are detected?

by hgsgm 4y ago

What would folks think of this compromise? Do the AI threat detection stuff on the device, and warn the user if suspected abusive images/text are detected?

olliej 4y ago

The governments don't want the user to be told, they want the government to be told. There are already numerous stories of people suffering tremendous harm because of the systems that are already in place for this purpose. Do you have a photo of your tiny kids playing in a pool? better hope google doesn't report you to the police and permanently ban you from all their services and delete all your data.

singleshot_ 4y ago
I think it’s far, far worse than this. Have you ever been inconvenienced because of a software bug? Now imagine you’re in jail for child sexual exploitation. Have you ever tried to contact one of these big service providers to get something fixed? Would you like Google tech support (snort) be the last thing standing between you and a prison sentence? I think the absolute last place we want to be is in a world where we can’t make money to eat without our portable computer/phone that runs broken software made by some scammy dot com whose salient feature is that if we don’t stay ahead of it, it will imprison the user.
- olliej 4y ago
  I agree entirely, but I used the google example because they've specifically done exactly that: * Reported an image for being CSAM * Terminated the account * The police correctly recognized it isn't CSAM * Google refused to reinstate the account But even if you assume a functioning and rule-of-law based legal system and you don't go to jail without actually committing a crime, the cost of bogus CSAM reports is in no way lost on me, especially in the conservative US and many similar regressive regimes around the world where LGBT anything is legislated as child abuse. Even if you are found innocent (which again, regressive regimes make hard) and news reports and anything on the internet will be "X was arrested for having child abuse imagery" nothing will ever be updated to reflect the result. This is ignoring what happens if people maliciously inject child abuse into your device's "CSAM" filter. Before the "CSAM" detection gets expanded to "weapons", "graffiti", "evidence of crime", etc. Take the UK where politicians aggressively passed laws to allow spying to prevent "terrorism", and those laws are used almost exclusively for councils to enforce local ordinances related to cleaning, taxes, and fees. And of course none of those laws were needed to prevent the crimes they're ostensibly meant to stop (literally multiple reports to the police prior to Manchester, etc).
throwaway67743 4y ago
There is a solution to the latter point, don't use malicious companies like Google, because really they're awful in so many other ways this is way down on the list.
- olliej 4y ago
  Who would you propose instead of google, given they'd all [in this scenario] be subject to the same laws regarding scanning and notification?
  - throwaway67743 4y ago
    You can host whatever it is yourself which exempts you from most obligations, and even if not you're free to ignore the rules because nobody cares about you as an individual. There are enough alternatives also these days that may not be obligated due to size or jurisdiction etc (eg incoming EU regulations require a minimum user count, and such) The USA is unlikely to blanket ban encryption due to the way the country is organised from a federal and state point of view, and others like the UK are basically irrelevant as nobody will rule in their favour over domestic entities, as much as they'd like to believe otherwise and many countries just do not care about anyone else so that is always an option.
    - olliej 4y ago
      The whole point of on device detection is that no hosting is involved or required: the media is scanned on device, and if the media matches the criminal material (actual child abuse, but in many US states easily anything transition or LGBT related, etc in other states anything gun or weapon related, etc) that is directly reported to law enforcement. Your ability to use an EU service gets exceedingly restricted when the US requires all EU businesses that serve US customers to comply with US law.
      - throwaway67743 4y ago
        Well sure but the parent comment was re Google, who do cloud scanning already, that can be avoided by doing it yourself - device scanning is pretty pointless as the people who it's "intended" to catch won't use devices that have that capability and more likely it'll just be used by the more authoritarian states to curb dissent etc, or put people in jail for having a meme with Hitler in it... The USA is still probably safe there, the EU do what they want anyway without any regard for the impact it has. (From the POV of an EU citizen with US legal entities, so I'm getting shafted from both sides)

tadfisher 4y ago

How about the users receiving abusive images/text report them to the government when received? Don't need to break E2EE for that.

Paul-Craft 4y ago
You're assuming the recipients don't want to receive said data.

singleshot_ 4y ago

I would think that’s a profoundly intrusive way to treat 330 million citizens of a liberal democracy against whom you have discovered no reasonable suspicion and developed no probable cause. We have a fourth amendment for a reason and it’s to prevent evil tyrants from oppressing us. Run whatever you want on your phone but this is a hard pass from me.

waselighis 4y ago

You may already be aware, but Apple tried to implement this on their devices a couple years ago. "Client side scanning". There was huge public backlash and Apple eventually moved away from the idea.