4 ms·

Google’s Scans of Private Photos Led to False Accusations of Child Abuse

59 points by colinprince 4y ago

alexb_ 4y ago
The blog post about the EU proposal is insanely scary. It "boasts" about 88% accuracy in "grooming" and 90% accuracy in new CSAM. And it also has this line, which is just so obviously horrendous: >the proposed obligations on service providers as regards the detection of CSAM are technologically neutral, meaning they do not prescribe which technology should be used for detection as long as the chosen technology meets the high standard of conditions and safeguards to protect all users. It is an obligation of result, not of means. Literally saying "The ends justify the means" when it comes to detecting CSAM. Remember, it NEVER has to do with protecting anyone. Governments do what gives them more power or they are already incentivized to do, and nothing else. And with no voter pressure against these types of laws, it's obvious what will happen. https://ec.europa.eu/commission/commissioners/2019-2024/johansson/blog/children-deserve-protection-and-privacy_en https://ec.europa.eu/commission/commissioners/2019-2024/joha... Here is the link to the blog post.
- isaacfrond 4y ago
  That is not what technologically neutral means. It means that the EU will not prescribe MS whatever or Google such and such, but that states can choose software themselves, assuming it works good enough (which apparently is not a very high bar to clear in this case).
  - raxxorraxor 4y ago
    How would you determine efficiency then? This will also only ever apply popular services. No random e2e encrypted messenger will implement this.
- account-5 4y ago
  > Literally saying "The ends justify the means" when it comes to detecting CSAM... I don't read it like that. My reading of it is that the EU don't care what technology is being used as long as the technology works as prescribed, that is, the result is CSAM detection regardless of the means by which it was detected. I don't read anything in that about government power or government overreach, regardless that these technologies are clearly very intrusive.
- bluesign 4y ago
  > It "boasts" about 88% accuracy in "grooming" and 90% accuracy in new CSAM Aside from principals, this is actually very nice accuracy technologically.
- ihaveajob 4y ago
  That seems like an uncharitable read. The phrase "an obligation of result, not of means" simply stating that the regulation doesn't choose a specific methodology, as it only prescribes a standard for its results. It's as if regulations for clinical thermometers require accuracy within a 0.1C margin of error, but manufacturers can choose how the device works, and how to make it. Of course, the scary part is that they would consider an 10% error rate to be OK.
  - alexb_ 4y ago
    In what universe should I be charitable to the government?
spaetzleesser 4y ago
Something definitely needs to be done about these big tech companies not having any appeals processes when they flag/suspend somebody. I know somebody whose business basically got shut down by Paypal and he never got an explanation or a way to rectify things. Makes me wonder more and more whether the tech companies are so profitable because of their tech or whether they are so profitable mainly because they cut corners by not hiring people who can deal with customer support. They are the perfect embodiment of faceless bureaucracy and Kafka-esque processes.
tantalor 4y ago
Previously: https://news.ycombinator.com/item?id=32538805 https://news.ycombinator.com/item?id=32538805
isaacfrond 4y ago
Just off the front page: Google refuses to reinstate account after man took medical images of son’s groin (theguardian.com) https://news.ycombinator.com/item?id=32560361 https://news.ycombinator.com/item?id=32560361
_8j50 4y ago
When their system flagged it as CP and their people reviewed it, how is that not a crime? In their state of mind they knew it was CP and they are not law enforcement or government lawyers. Civilians at the behest of google watched CP amd there was no prosecution! Isn't that another big story here? I absolutley think CSAM as an idea is a good thing so long as any fault in the algorithms are taken into account and it is law enforcement that does the investigation begining to end. Google has the right to refuse service, but not with cause. They slandered a person and cancelled service before the persons had time to migrate causing considerable material damage. How much monetary value do you place on sentimental pictures or email messages? If money is an issue let us donate to the legal funds of these people but they should absolutley sue the crap out of google. I don't know if there is a civil cause for action in any CP related law but if google employees who are not law enfrocement or authorized medical professionals saw the private pictures of my child I would be raising hell in form of lawsuits and in every avenue where I can make this known including election campaigns against the DA that didn't prosecute google employees and execs for intentional viewing of CP. You don't like CSAM as a concept? Ok, I get the arguments against it but laws are too humane against child predators. Until castrations , public executions and death by torture are on the books (prevention) then CSAM like approach is a must (detection). Refusing to take a strong stance on prevention and on detection is enabling the real predators.
- account-5 4y ago
  Are you suggesting law enforcement personnel should be embedded in Google to do the confirmation of CSAM? Someone has to confirm CSAM and report to the police. Or would you rather an auto-report directly to the police from the algorithm that flagged it with no human oversight?
  - _8j50 4y ago
    Auto report to police when there is algorithmic match then the police do their own vetting and decide to open an investigation,prosecute,etc... no punitive action by google until the police decide to prosecute. Google employess looking at cp is a crime since they are not legally authorized or perimitted by the parents to look at the pictures in any capacity. LE/gov also has no authority to delegate this task to undeputized civilians.
rickspencer3 4y ago
I read the original NYT article as well. I disagree with the position that the "AI failed." I suspect that the images were pretty much dead on for suspected child abuse image from how they were described in the original article. What I think is terrible about the story is that Google seemed supremely uninterested in investing any effort whatsoever in helping their customers recover from the false allegations. The customers ended up losing all of the information that they had stored on Google, and the company seemed to treat exoneration of the hideous crime that they accused the customers of to be a triviality not worth their effort. I am a big fan of Google's technology, and I am deeply wedged into their ecosystem for my personal life (Android, Google Maps, Google Photos, Gmail, etc...). I even pay for premium services. But, it's clear from how they treat Youtube creators and stories like this that, as a customer, they are wholly indifferent to customers as individuals. You have to play defense against their tech.
- account-5 4y ago
  I don't trust Google with anything that I need to keep either private or access too. The company doesn't care about their "customers" because their actual customers are the advertising companies they sell their "customers" data to. Google has nothing of importance from me because they are unreliable and untrustworthy.
  - candiodari 4y ago
    Don't you think that the actions of governments have a lot more to do with what happened here than Google's own policies? 1) Google is forced to do CSAM scanning and report any offending material to authorities by EU 2) they are forced to close accounts where they find this account 3) they are forced to delete any accounts so closed by GPDR regulations This isn't Google, Facebook or LinkedIn doing this, the only thing Google has to do with this is that they're big enough to be one of the first targets of police forces of multiple nations to implement this. https://en.wikipedia.org/wiki/EARN_IT_Act https://en.wikipedia.org/wiki/EARN_IT_Act https://european-pirateparty.eu/chatcontrol-commission-presents-mass-surveillance-plan-may-11/ https://european-pirateparty.eu/chatcontrol-commission-prese... https://en.wikipedia.org/wiki/General_Data_Protection_Regulation https://en.wikipedia.org/wiki/General_Data_Protection_Regula... How the police actually deals with these companies: https://www.zdnet.com/article/signal-unveils-how-far-us-law-enforcement-will-go-to-get-information-about-people/ https://www.zdnet.com/article/signal-unveils-how-far-us-law-... "Though the judge approved four consecutive non-disclosure orders, the court never acknowledged receipt of our motion to partially unseal, nor scheduled a hearing, and would not return counsel's phone calls seeking to schedule a hearing," Signal wrote. Oh ... and of course, now many US states have decided that for the police's sacred duty to prevent abortions, the police do NOT have to respect their jurisdiction (they get to attack people for crimes, proven ... or not, committed elsewhere when related to abuse, like CSAM)... Sorry, but the problem here is not tech companies, except in the sense that you may be more or less exposed to this problem depending on how much private info you store with them.
engineer_dude 4y ago
This is why I don't have a Dropbox account anymore. I am extremely fortunate that the account that was deleted without recourse only contained data I had copies of on my hard drive, and to my knowledge law enforcement isn't involved. The article fails to mention the stress and trauma of being accused of having CSAM. That remains to this day ... I'm posting from an alt because even the false accusation carries a potentially career and family destroying stigma.
ytch 4y ago
I'd use VeraCrypt along with those CSAM enabled cloud, so they cannot read my files in most cases. (unless they assume my file is so valuable that an exploit is needed to decrypt them)
cassianoleal 4y ago
This should read as "Google's Scans of Photos Uploaded by User (...)".

59 points by colinprince 4y ago

The blog post about the EU proposal is insanely scary. It "boasts" about 88% accuracy in "grooming" and 90% accuracy in new CSAM. And it also has this line, which is just so obviously horrendous: >the proposed obligations on service providers as regards the detection of CSAM are technologically neutral, meaning they do not prescribe which technology should be used for detection as long as the chosen technology meets the high standard of conditions and safeguards to protect all users. It is an obligation of result, not of means. Literally saying "The ends justify the means" when it comes to detecting CSAM. Remember, it NEVER has to do with protecting anyone. Governments do what gives them more power or they are already incentivized to do, and nothing else. And with no voter pressure against these types of laws, it's obvious what will happen. https://ec.europa.eu/commission/commissioners/2019-2024/johansson/blog/children-deserve-protection-and-privacy_en https://ec.europa.eu/commission/commissioners/2019-2024/joha... Here is the link to the blog post.

isaacfrond 4y ago
That is not what technologically neutral means. It means that the EU will not prescribe MS whatever or Google such and such, but that states can choose software themselves, assuming it works good enough (which apparently is not a very high bar to clear in this case).
- raxxorraxor 4y ago
  How would you determine efficiency then? This will also only ever apply popular services. No random e2e encrypted messenger will implement this.
account-5 4y ago
> Literally saying "The ends justify the means" when it comes to detecting CSAM... I don't read it like that. My reading of it is that the EU don't care what technology is being used as long as the technology works as prescribed, that is, the result is CSAM detection regardless of the means by which it was detected. I don't read anything in that about government power or government overreach, regardless that these technologies are clearly very intrusive.
bluesign 4y ago
> It "boasts" about 88% accuracy in "grooming" and 90% accuracy in new CSAM Aside from principals, this is actually very nice accuracy technologically.
ihaveajob 4y ago
That seems like an uncharitable read. The phrase "an obligation of result, not of means" simply stating that the regulation doesn't choose a specific methodology, as it only prescribes a standard for its results. It's as if regulations for clinical thermometers require accuracy within a 0.1C margin of error, but manufacturers can choose how the device works, and how to make it. Of course, the scary part is that they would consider an 10% error rate to be OK.
- alexb_ 4y ago
  In what universe should I be charitable to the government?

spaetzleesser 4y ago

Something definitely needs to be done about these big tech companies not having any appeals processes when they flag/suspend somebody. I know somebody whose business basically got shut down by Paypal and he never got an explanation or a way to rectify things. Makes me wonder more and more whether the tech companies are so profitable because of their tech or whether they are so profitable mainly because they cut corners by not hiring people who can deal with customer support. They are the perfect embodiment of faceless bureaucracy and Kafka-esque processes.

tantalor 4y ago

Previously: https://news.ycombinator.com/item?id=32538805 https://news.ycombinator.com/item?id=32538805

isaacfrond 4y ago

Just off the front page: Google refuses to reinstate account after man took medical images of son’s groin (theguardian.com) https://news.ycombinator.com/item?id=32560361 https://news.ycombinator.com/item?id=32560361

_8j50 4y ago

When their system flagged it as CP and their people reviewed it, how is that not a crime? In their state of mind they knew it was CP and they are not law enforcement or government lawyers. Civilians at the behest of google watched CP amd there was no prosecution! Isn't that another big story here? I absolutley think CSAM as an idea is a good thing so long as any fault in the algorithms are taken into account and it is law enforcement that does the investigation begining to end. Google has the right to refuse service, but not with cause. They slandered a person and cancelled service before the persons had time to migrate causing considerable material damage. How much monetary value do you place on sentimental pictures or email messages? If money is an issue let us donate to the legal funds of these people but they should absolutley sue the crap out of google. I don't know if there is a civil cause for action in any CP related law but if google employees who are not law enfrocement or authorized medical professionals saw the private pictures of my child I would be raising hell in form of lawsuits and in every avenue where I can make this known including election campaigns against the DA that didn't prosecute google employees and execs for intentional viewing of CP. You don't like CSAM as a concept? Ok, I get the arguments against it but laws are too humane against child predators. Until castrations , public executions and death by torture are on the books (prevention) then CSAM like approach is a must (detection). Refusing to take a strong stance on prevention and on detection is enabling the real predators.

account-5 4y ago
Are you suggesting law enforcement personnel should be embedded in Google to do the confirmation of CSAM? Someone has to confirm CSAM and report to the police. Or would you rather an auto-report directly to the police from the algorithm that flagged it with no human oversight?
- _8j50 4y ago
  Auto report to police when there is algorithmic match then the police do their own vetting and decide to open an investigation,prosecute,etc... no punitive action by google until the police decide to prosecute. Google employess looking at cp is a crime since they are not legally authorized or perimitted by the parents to look at the pictures in any capacity. LE/gov also has no authority to delegate this task to undeputized civilians.

rickspencer3 4y ago

I read the original NYT article as well. I disagree with the position that the "AI failed." I suspect that the images were pretty much dead on for suspected child abuse image from how they were described in the original article. What I think is terrible about the story is that Google seemed supremely uninterested in investing any effort whatsoever in helping their customers recover from the false allegations. The customers ended up losing all of the information that they had stored on Google, and the company seemed to treat exoneration of the hideous crime that they accused the customers of to be a triviality not worth their effort. I am a big fan of Google's technology, and I am deeply wedged into their ecosystem for my personal life (Android, Google Maps, Google Photos, Gmail, etc...). I even pay for premium services. But, it's clear from how they treat Youtube creators and stories like this that, as a customer, they are wholly indifferent to customers as individuals. You have to play defense against their tech.

account-5 4y ago
I don't trust Google with anything that I need to keep either private or access too. The company doesn't care about their "customers" because their actual customers are the advertising companies they sell their "customers" data to. Google has nothing of importance from me because they are unreliable and untrustworthy.
- candiodari 4y ago
  Don't you think that the actions of governments have a lot more to do with what happened here than Google's own policies? 1) Google is forced to do CSAM scanning and report any offending material to authorities by EU 2) they are forced to close accounts where they find this account 3) they are forced to delete any accounts so closed by GPDR regulations This isn't Google, Facebook or LinkedIn doing this, the only thing Google has to do with this is that they're big enough to be one of the first targets of police forces of multiple nations to implement this. https://en.wikipedia.org/wiki/EARN_IT_Act https://en.wikipedia.org/wiki/EARN_IT_Act https://european-pirateparty.eu/chatcontrol-commission-presents-mass-surveillance-plan-may-11/ https://european-pirateparty.eu/chatcontrol-commission-prese... https://en.wikipedia.org/wiki/General_Data_Protection_Regulation https://en.wikipedia.org/wiki/General_Data_Protection_Regula... How the police actually deals with these companies: https://www.zdnet.com/article/signal-unveils-how-far-us-law-enforcement-will-go-to-get-information-about-people/ https://www.zdnet.com/article/signal-unveils-how-far-us-law-... "Though the judge approved four consecutive non-disclosure orders, the court never acknowledged receipt of our motion to partially unseal, nor scheduled a hearing, and would not return counsel's phone calls seeking to schedule a hearing," Signal wrote. Oh ... and of course, now many US states have decided that for the police's sacred duty to prevent abortions, the police do NOT have to respect their jurisdiction (they get to attack people for crimes, proven ... or not, committed elsewhere when related to abuse, like CSAM)... Sorry, but the problem here is not tech companies, except in the sense that you may be more or less exposed to this problem depending on how much private info you store with them.

engineer_dude 4y ago

This is why I don't have a Dropbox account anymore. I am extremely fortunate that the account that was deleted without recourse only contained data I had copies of on my hard drive, and to my knowledge law enforcement isn't involved. The article fails to mention the stress and trauma of being accused of having CSAM. That remains to this day ... I'm posting from an alt because even the false accusation carries a potentially career and family destroying stigma.

ytch 4y ago

I'd use VeraCrypt along with those CSAM enabled cloud, so they cannot read my files in most cases. (unless they assume my file is so valuable that an exploit is needed to decrypt them)

cassianoleal 4y ago

This should read as "Google's Scans of Photos Uploaded by User (...)".