7 ms·
How we detect, remove and report child sexual abuse material
- dcwardell 4y agoThey mention suspending accounts responsible for uploading CSAM. I'm curious what their process looks like when their own bots crawl a domain, find the CSAM, and then make it available to users via their image search. What action is taken against domains hosting CSAM? How quickly do they remove the content from search results, etc?
- notch656a 4y agoThey also conveniently forward to law enforcement parents who send non-sexual explicit images taken expressly for the purpose of examination by licensed medical personnel. https://www.theguardian.com/technology/2022/aug/22/google-csam-account-blocked https://www.theguardian.com/technology/2022/aug/22/google-cs...
- danpalmer 4y agoI've read a bunch of articles about this and I just can't see how there isn't more to the story. Whether the original ban was automated or not, one of their specialists must have been involved by now, not least because of the media coverage and the fact they reported it to the police. While the police declined to investigate further (from what we know, last time I read about this), there must be another reason for the account ban to remain in place. I'm biased, I work at Google (I don't know anything about this that hasn't been reported), but I also just don't believe that a company would wilfully ban accounts that they don't have a good reason to ban, after human review. It would be a stupid move, especially considering the media attention. Hence my guess that there must be more to this story. Again I have no insider knowledge, this is purely personal speculation.
- dclusin 4y agoMultiple independent organizations inside a company each with their own rigid and inflexible operating procedures. The user to moderator ratio in the tens of thousands to 1. How is this hard to see?
- kahrl 4y ago"I'm biased, I work at Google"
- danpalmer 4y agoI don't know what you mean by the multiple organizations thing – this seems to be squarely in the realm of trust and safety. And while I understand the frustration in contacting a human sometimes, in this case it seems that a human will have reviewed the content and come to the conclusion based on their training, expert opinions, policies, etc (as per the article). What can human contact add here? Excuses or reasoning for the content? Given the media attention, someone will almost certainly have reached out if there was any way that a reason or excuse could possibly change the outcome. If they didn't that implies again that there's more to this story. I'm not going to pass any judgement on what the "more to the story" is here, I don't have all the facts, but it seems obvious to me that there are more facts, and that people are working from different amounts of information here.
- dclusin 4y agoThe two pieces of information I think that stand out are: 1. Law enforcement elected not to pursue charges 2. The banned parties were comfortable enough going to a respected newspaper with the facts of their case, which, the newspaper presumably also did due diligence and sided with the users. I don’t think that there’s really any “hidden” context here that would convince many (except maybe those who get their paychecks from big G) that google is not in the wrong. Perhaps the hidden context here is that they also investigated the couple after the journal report and found other “violations” which were ban worthy after, not before, their case was aired publicly. I don’t think many people would consider this theoretical scenario fair either.
- 4y ago
- wmf 4y agoPolicy #0: Never admit a mistake.
- desmosxxx 4y agoThere are possible legal reasons they wouldn't reverse an account ban of this sort. There are also several instances of Google employees or their spouses being banned and AFAIK none have led to any criminal charges - e.g. https://twitter.com/miguelytob/status/1315749803041619981 https://twitter.com/miguelytob/status/1315749803041619981 (not sure if this one was for csam or not - but IIRC some were). Police salivate over cases like this - until they decide to prosecute, it seems obvious that Google is in the wrong here.
- lupire 4y agoNo large business has ever said "I'll let my customers do whatever they want until the police forcibly stop them". It's too risky and unsavory. They set up a buffer around the law. Even legal adult porn is enough to get in trouble with Google.
- saurik 4y ago> It would be a stupid move... Please do remember that this company you work for is somewhat notorious on this website for making "stupid moves" seemingly every other day with respect to basic product management. The usual defense is that some internal incentive mechanism blindly optimized for these stupid moves and no one is at the wheel to change the company's high-level behavior... is there really any reason to believe things are different here? Is there a mechanism by which someone helping a user get around a CSAM filter would get them promoted at Google, or would they just be taking on potentially-arbitrary levels of internal risk going off the rails on a side project wherein they have to condone something that has an air of danger to it with the only real goal of helping a user who probably wasn't even paying Google anything in the first place? I admittedly don't work at Google, but the impression one gets on the outside looking in is that doing something that wouldn't cause some major impact on your performance review "isn't very Googley" :(.
- dclusin 4y agoAnd don’t restore accounts even when law enforcement decides not to pursue charges.
- 7952 4y agoThat is expecting Google to prove a negative about the user. They probably don't want to have the obligation to verify that the user really is innocent. And police inaction is not evidence of that.
- deleted 4y ago[deleted]
- boredumb 4y agoI don't envy whoever had to aggregate the training data.
- kazinator 4y ago> This combination enables us to detect CSAM on our platforms at scale, while keeping our false positive rate extremely low. Yeah, right! https://www.theguardian.com/technology/2022/aug/22/google-csam-account-blocked https://www.theguardian.com/technology/2022/aug/22/google-cs...
- Spivak 4y agoPeople really need to understand that small percent times big number is a normal sized number. And you are pointing out one "false positive" that actually wasn't a false positive at all. You're not gonna be able to distinguish pictures of a naked child taken for acceptable reasons and not. The story is Google not reinstating the account when the police said it was kosher which is a human problem.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- nigrioid 4y agoGreat! If they decide someone has to go, all they have to do is say they found CP on their Google Drive. Who is going to believe the person when they say it was planted? If you're having a hard time assessing the risk, imagine Trump was back in office and he was out to get all of the valiant anti-fascists.
- ruune 4y agoSame with any other company that hosts files for you. Microsoft, GitHub, Dropbox, your email host, your server host, etc.
- deleted 4y ago[deleted]
- jasonjei 4y agoThis doesn’t address the issue of banning and locking a parent out of a Google account from sending a photo to a pediatric medical provider during Covid telehealth: https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?unlocked_article_code=AAAAAAAAAAAAAAAACEIPuonUktbfqYhkQFUZAibJUNMnqBqCgvfeh7M7gXD6IW2OVCxcwuURB4GG8EqIea1va98-xXGDRNhdN_kvWPl2hKd5DnBadjOJ8NGCiYhXZGI8s56yVWc7mJmXAr02rTK2Kze1dbw6me3vtFfWbmfpRKXU2CUjJQ1lqJBuIUq4miBdntezGeV229V-3_wnF8o6EW9GPH_WyqGuXxZuO9yGbQXf6x02UoxaUDLVm92d49EQYVkYSAKGHD4kvzFKuJ4LM8gXPa3_MxYjZML95L0bAWFtJ4qrbIYjCOYN6VHrivKhqQVp5Xv9cQ&smid=nytcore-ios-share&referringSource=articleShare https://www.nytimes.com/2022/08/21/technology/google-surveil...
- deleted 4y ago[deleted]
- concinds 4y agoIt's sad that this doesn't get the same attention as when Apple tried something similar. Google's not using PhotoDNA (a preexisting hash-based list). They're using flawed image recognition algorithms to try to recognize never-seen-before CSAM. Apple's proposal was actually far better for privacy than this; and though it was done on-device, it had a threshold of 30 images. Google is doing far worse than Apple proposed, and nobody seems to care.
- greatNespresso 4y ago"We also have a team of highly specialized and trained content reviewers and subject matter experts who help ensure that our technology delivers accurate results" Wondering what does the training look like.
- lupire 4y agoFor your own sake, don't try to find out.
- christkv 4y agoMust be literally the worst and most traumatic job I can think of in tech.
- danpalmer 4y ago> Our team members bring deep expertise to this work with backgrounds in law, child safety and advocacy, social work, and cyber investigations, among other disciplines. They are specially trained on both our policy scope and what legally constitutes child sexual abuse material. We regularly update this training and our guidelines in consultation with legal counsel, independent experts and medical professionals. Sounds pretty thorough and covers all the bases that I'd expect for this sort of thing.
- Hizonner 4y agoIf they are manually verifying all of the hashes, and training ML to detect previously unreported images, that means they must be retaining a vast collection of material that they know is illegal. Not just passing it on with reports, but intentionally collecting it, retaining it for the long term, and processing it. Anybody have any idea why they're allowed to do that?
- danielvaughn 4y agoI’d be surprised if storing the hashes is illegal.
- alach11 4y agoThe hashes are fine. But to train image recognition models they’re going to need to retain the original images.
- nigrioid 4y agoThey are allowed to do it because it benefits "law enforcement". See my other comment.
- DoItToMe81 4y agoMost countries have legal clauses permitting storage for filter work.
- lupire 4y agoNothing in the article implies what you suggest. There is plenty of legal porn imagery and legal non porn imagery and legal child imagery and legal adult imagery. It's not so hard to make your system detect if an image is similar to both porn and child.
- vinni2 4y agoNothing in the article implies what you suggest.
- 4y ago
- lupire 4y agoWhat a weird non-article. No "how", just "yeah, we check for known bad files, and AI magic"
- deleted 4y ago[deleted]
- eimrine 4y agoIs it possible to make Google's filter to get false possible? I think if yes then it can be malused, similarly to incident with father and his son's medical image of groin.
- qull 4y ago
- BXLE_1-1-BitIs1 4y agoCompletely missing from this blog post are: 1. Instructions on how to legitimately share photos with medical providers. 2. A transparent appeal process 3. An address for legal service. The family that sent a photo of their toddler's penile lesion to their pediatrician is still left in the ditch. Will their ever be an apology and reinstatement for this family?
- T3OU-736 4y agoWhile I wholeheartedly agree that Google should reverse the their actions in this case, I would argue that Google Drive/Photos is not the right "product" for medical (so, in the US, HIPPA-protected) data exchange.
- BXLE_1-1-BitIs1 4y agoThe Google default is that your photos get automatically uploaded. And then it's Bambi meets Godzilla. Google's AI would do better to refuse to upload and notify the user that the photo may be CSAM - AND give the user an opportunity to reply to a human AND explain the appeal process.
- GaryNumanVevo 4y agoI feel like we'll never hear the full details of that story. It seems like a pretty easy thing to fix from Google's side. However, I think it's telling that he still remains banned and Google hasn't made any public statement. It feels like a constructed narrative piece. It's definitely a possibility that the father had CSAM in his possession and decided to go to the media, trying to get his account back, knowing full well that Google can't confirm or deny it publicly.
- greatgib 4y agoThis is just a great reminder that the cloud is just someone else computer! Now tech companies are doing justice themselves. I understand the good purpose for why they want to do that. Almost no one will contest the good side of that work. But, think about it, if for the "greater good", they were to put camera+mic everywhere in your house and say that it is legitimate to monitor your home 24/24 for the occasional case you might be part of the minority of child abusers or, in the current trend, something similar like having hate speech at home and propagating fake news to your friends and family? This is how you finish with the 1984 world. Even if being 'suspicious' of any individual by default without even a good cause, is like wiretapping the entire population by default in my opinion.