5 ms·

Stratechery covered this … they pointed out the really f*ed: part: Google is refusing to reinstate the account. Protecting children is important. AI is imperfe

Stratechery covered this … they pointed out the really f*ed: part: Google is refusing to reinstate the account. Protecting children is important. AI is imperfect. But there is no reason the keep the account suspended once it’s clear there was no wrongdoing. This man is innocent of doing anything wrong. Google had suspended him, and removed access to all his online account data. And refuses to reinstate. We balance out liberties with responsibilities all the time. We allow the state power to protect children, and corporations have the right to assist them. There are careful balancing acts being done. But there is no balancing act here. There is no justification for Googles action.

gjsman-1000 4y ago
This is the part that should receive a Congressional inquiry. I would love to see Congress drag Sundar Pichai (and no one else) before them and scream “What the hell is wrong with you?” in his face. We have courts for a reason. They found no wrongdoing. Google is objectively deciding they know better than an elected court on what happened. This whole situation is absurd and evil.
- lrvick 4y ago
  Remember Google can ban any account at any time for any reason including because they just do not like you. No Google software, devices, or data on their systems belongs to you. Sue them and they will remind you of this fact in detail. If you want to own your digital life you will have to commit to being a pariah in your social circle that is ridiculed constantly for using open source tools.
  - rcoveson 4y ago
    > If you want to own your digital life you will have to commit to being a pariah in your social circle that is ridiculed constantly for using open source tools. I think your friends are just a little too obsessed with social media. The only response I ever hear to "I don't have a facebook/twitter/whatever" is "yeah, that makes sense, I think about deleting mine all the time."
- josephcsible 4y ago
  > Sundar Pichai (and no one else) I'd go a little broader than that. I'd also want to see them drag in whoever made the decision to not reinstate the account, plus everyone who knew what happened and could have overridden the decision but chose not to do so.
  - AlbertCory 4y ago
    > everyone who knew what happened and could have overridden the decision that will never happen, unless documents & emails get discovered and all the cc'ed people get subpoena'ed and deposed. And even that won't catch everybody. Congressional committees don't have time for that. Big corporations are adept at diffusing real responsibility among a faceless mass of people.
gonehome 4y ago
IIRC there was a follow up which was arguably worse. Google implied that the account was suspended because of real CSAM concerns unrelated to the photos sent to the doctor/police officer (other photos in their account? they wouldn't go into details about their decision) and that the officer closing the case doesn't mean their judgement here is wrong. If that's the case I can understand why they're obstinate about their decision (which otherwise would seem like a dumb mistake they should just reverse), but the problem is none of this happens in a place where users have any ability to get reinstated or have any sort of control over their digital life - there's no real path any individual has out of this even after going to the press. There's no due process, no way to defend yourself, no way to get them to show how they made their decision. As I understand it you have no rights beyond the ToS. The user also losing all related account access (two factor, email, etc.) is particularly bad. This is also categorically different from Apple's approach which compared hashes of CSAM with what's in the NCMEC database which would not have caused this mistake (here Google is using computer vision to discover and flag novel images).
- trasz2 4y ago
  There is no reason for Google to not be lying about it.
- skissane 4y ago
  > There's no due process, no way to defend yourself, no way to get them to show how they made their decision. As I understand it you have no rights beyond the ToS. This is where legislation is required. Mass-scale social media/cloud providers/etc are effectively public utilities, and you should be able to challenge their decisions in court - the current situation is as bad as if the electricity company could disconnect you for non-specific reasons and you had no reasonable prospects of a successful legal challenge to their decision. (Such regulation should be limited to providers above a certain size-maybe a cutoff like 10 million MAU or 500 million annual revenue-so small players aren’t burdened by it.)
  - deleted 4y ago
    [deleted]
shadowgovt 4y ago
I'm not normally one to indulge conspiracy theories. ... But given the situation and high profile nature of this incident? If that account's still locked, it's locked under sealed FBI warrant. Google has had situations where they work hand-in-give with law enforcement to resolve something, and when they do, they're radio-silent on the situation. Sometimes for years, given the scope.
annoyingnoob 4y ago
This is what software as law looks like. The software actually found one or more pictures of a nude child and banned the account - the software worked exactly as intended. As humans we can see the issue here, the software cannot see or understand the issue. Google is not about to start carving exceptions out of its government mandated or coerced scanning feature, it can only follow the rule that any nudity involving children is bad.
- _aavaa_ 4y ago
  Expect that the software did not work as intended. It’s intended to identify child sexual abuse material and this was not that.
  - annoyingnoob 4y ago
    This is one group that Google works with: https://www.missingkids.org/theissues/csam https://www.missingkids.org/theissues/csam Their definition and yours are not exactly the same. If you read what they have to say, you'll see that the issues are more broad than whatever you personally call abuse, and those issues include exploitation and distribution of images that victims find harmful. Are you a victim of abuse or exploitation? Who are you to determine what a victim finds abusive or exploitative? I'm not saying I agree with this situation, just that I can see how it happened and why Google won't do anything in this case.
    - _aavaa_ 4y ago
      I didn't give a definition for what constitutes abuse. My point is that the software is not working as intended. The intent is to stop child abuse, but the software they build does not identify child abuse it identifies what it believes to be naked (semi-naked?) minors. Those are two different things. A photo you take of your kid for the doctor to see in order to treat them is not CSAM. Neither is a photo of your naked baby. But those same photos if stolen from you or taken by others for different (less wholesome) reasons would be. But, and most importantly, that context is not to be found in the photo itself.
wvenable 4y ago
> But there is no reason the keep the account suspended once it’s clear there was no wrongdoing. For you, Google is your everything. For Google, you're an tiny ant. If a tiny ant steps out of line, they kill it. They don't feel bad about the ant; they don't feel anything. It's a free service and it's not worth the cost to literally do anything.
867-5309 4y ago
>Protecting children is important. AI is imperfect. is protecting children more important than a software engineer with no backup's mistakes and first world inconveniences? yes. the AI did its job flawlessly - detected a naked toddler. as did the human verifier. bravo. should it have been detected and flagged? yes. should we be surprised at the chosen free cloud provider's attitude or dismissal? no.
- watwut 4y ago
  Maybe naked toddlers should not be flagged? You can see naked toddlers bathing in lakes and such. They are naked, cause overwhelming majority if people dont see toddlers as sexual and they themselves are not ashamed of body yet. Naked pre-teen? Sure flag absolutely. Toddler? No.
  - rmbyrro 4y ago
    They're not looking for "normal people", hence why a naked toddler is flagged.
- 2OEH8eoCRo0 4y ago
  CSAM stands for Child Sexual Abuse Material. It was not CSAM, quite the opposite actually.
  - 867-5309 4y ago
    thanks. so CSAs have no interest in pictures of naked toddlers..
einpoklum 4y ago
> Protecting children is important. Google protecting children is not important. > This man is innocent of doing anything wrong He is not entirely without fault: He negligently enabled surveillance of his personal files, including allowing Google to make copies of his son's genitalia / his son's private personal medicall information. He's not exactly innocent - but of course, his misdeed is not vis-a-vis the state or Google, but vis-a-vis his son, his wife and himself. > We allow the state power to protect children You don't allow the state anything. The state allows itself and you pretend we control the state because we get to make some choices via elections every once in a while.
deleted 4y ago
[deleted]

gjsman-1000 4y ago

This is the part that should receive a Congressional inquiry. I would love to see Congress drag Sundar Pichai (and no one else) before them and scream “What the hell is wrong with you?” in his face. We have courts for a reason. They found no wrongdoing. Google is objectively deciding they know better than an elected court on what happened. This whole situation is absurd and evil.

lrvick 4y ago
Remember Google can ban any account at any time for any reason including because they just do not like you. No Google software, devices, or data on their systems belongs to you. Sue them and they will remind you of this fact in detail. If you want to own your digital life you will have to commit to being a pariah in your social circle that is ridiculed constantly for using open source tools.
- rcoveson 4y ago
  > If you want to own your digital life you will have to commit to being a pariah in your social circle that is ridiculed constantly for using open source tools. I think your friends are just a little too obsessed with social media. The only response I ever hear to "I don't have a facebook/twitter/whatever" is "yeah, that makes sense, I think about deleting mine all the time."
josephcsible 4y ago
> Sundar Pichai (and no one else) I'd go a little broader than that. I'd also want to see them drag in whoever made the decision to not reinstate the account, plus everyone who knew what happened and could have overridden the decision but chose not to do so.
- AlbertCory 4y ago
  > everyone who knew what happened and could have overridden the decision that will never happen, unless documents & emails get discovered and all the cc'ed people get subpoena'ed and deposed. And even that won't catch everybody. Congressional committees don't have time for that. Big corporations are adept at diffusing real responsibility among a faceless mass of people.

gonehome 4y ago

IIRC there was a follow up which was arguably worse. Google implied that the account was suspended because of real CSAM concerns unrelated to the photos sent to the doctor/police officer (other photos in their account? they wouldn't go into details about their decision) and that the officer closing the case doesn't mean their judgement here is wrong. If that's the case I can understand why they're obstinate about their decision (which otherwise would seem like a dumb mistake they should just reverse), but the problem is none of this happens in a place where users have any ability to get reinstated or have any sort of control over their digital life - there's no real path any individual has out of this even after going to the press. There's no due process, no way to defend yourself, no way to get them to show how they made their decision. As I understand it you have no rights beyond the ToS. The user also losing all related account access (two factor, email, etc.) is particularly bad. This is also categorically different from Apple's approach which compared hashes of CSAM with what's in the NCMEC database which would not have caused this mistake (here Google is using computer vision to discover and flag novel images).

trasz2 4y ago
There is no reason for Google to not be lying about it.
skissane 4y ago
> There's no due process, no way to defend yourself, no way to get them to show how they made their decision. As I understand it you have no rights beyond the ToS. This is where legislation is required. Mass-scale social media/cloud providers/etc are effectively public utilities, and you should be able to challenge their decisions in court - the current situation is as bad as if the electricity company could disconnect you for non-specific reasons and you had no reasonable prospects of a successful legal challenge to their decision. (Such regulation should be limited to providers above a certain size-maybe a cutoff like 10 million MAU or 500 million annual revenue-so small players aren’t burdened by it.)
- deleted 4y ago
  [deleted]

shadowgovt 4y ago

I'm not normally one to indulge conspiracy theories. ... But given the situation and high profile nature of this incident? If that account's still locked, it's locked under sealed FBI warrant. Google has had situations where they work hand-in-give with law enforcement to resolve something, and when they do, they're radio-silent on the situation. Sometimes for years, given the scope.

annoyingnoob 4y ago

This is what software as law looks like. The software actually found one or more pictures of a nude child and banned the account - the software worked exactly as intended. As humans we can see the issue here, the software cannot see or understand the issue. Google is not about to start carving exceptions out of its government mandated or coerced scanning feature, it can only follow the rule that any nudity involving children is bad.

_aavaa_ 4y ago
Expect that the software did not work as intended. It’s intended to identify child sexual abuse material and this was not that.
- annoyingnoob 4y ago
  This is one group that Google works with: https://www.missingkids.org/theissues/csam https://www.missingkids.org/theissues/csam Their definition and yours are not exactly the same. If you read what they have to say, you'll see that the issues are more broad than whatever you personally call abuse, and those issues include exploitation and distribution of images that victims find harmful. Are you a victim of abuse or exploitation? Who are you to determine what a victim finds abusive or exploitative? I'm not saying I agree with this situation, just that I can see how it happened and why Google won't do anything in this case.
  - _aavaa_ 4y ago
    I didn't give a definition for what constitutes abuse. My point is that the software is not working as intended. The intent is to stop child abuse, but the software they build does not identify child abuse it identifies what it believes to be naked (semi-naked?) minors. Those are two different things. A photo you take of your kid for the doctor to see in order to treat them is not CSAM. Neither is a photo of your naked baby. But those same photos if stolen from you or taken by others for different (less wholesome) reasons would be. But, and most importantly, that context is not to be found in the photo itself.

wvenable 4y ago

> But there is no reason the keep the account suspended once it’s clear there was no wrongdoing. For you, Google is your everything. For Google, you're an tiny ant. If a tiny ant steps out of line, they kill it. They don't feel bad about the ant; they don't feel anything. It's a free service and it's not worth the cost to literally do anything.

867-5309 4y ago

>Protecting children is important. AI is imperfect. is protecting children more important than a software engineer with no backup's mistakes and first world inconveniences? yes. the AI did its job flawlessly - detected a naked toddler. as did the human verifier. bravo. should it have been detected and flagged? yes. should we be surprised at the chosen free cloud provider's attitude or dismissal? no.

watwut 4y ago
Maybe naked toddlers should not be flagged? You can see naked toddlers bathing in lakes and such. They are naked, cause overwhelming majority if people dont see toddlers as sexual and they themselves are not ashamed of body yet. Naked pre-teen? Sure flag absolutely. Toddler? No.
- rmbyrro 4y ago
  They're not looking for "normal people", hence why a naked toddler is flagged.
2OEH8eoCRo0 4y ago
CSAM stands for Child Sexual Abuse Material. It was not CSAM, quite the opposite actually.
- 867-5309 4y ago
  thanks. so CSAs have no interest in pictures of naked toddlers..

einpoklum 4y ago

> Protecting children is important. Google protecting children is not important. > This man is innocent of doing anything wrong He is not entirely without fault: He negligently enabled surveillance of his personal files, including allowing Google to make copies of his son's genitalia / his son's private personal medicall information. He's not exactly innocent - but of course, his misdeed is not vis-a-vis the state or Google, but vis-a-vis his son, his wife and himself. > We allow the state power to protect children You don't allow the state anything. The state allows itself and you pretend we control the state because we get to make some choices via elections every once in a while.

deleted 4y ago

[deleted]