7 ms·

A Developer Accidentally Found CSAM in AI Data. Google Banned Him for It

122 points by markatlarge 10mo ago

bsowl 10mo ago
More like "A developer accidentally uploaded child porn to his Google Drive account and Google banned him for it".
- jkaplowitz 10mo ago
  The penalties for unknowingly possessing or transmitting child porn are far too harsh, both in this case and in general (far beyond just Google's corporate policies). Again, to avoid misunderstandings, I said unknowingly - I'm not defending anything about people who knowingly possess or traffic in child porn, other than for the few appropriate purposes like reporting it to the proper authorities when discovered.
  - jjk166 10mo ago
    The issue is that when you make ignorance a valid defense, the optimal strategy is to deliberately turn a blind eye, as it reduces your risk exposure. It further gives refuge for those who can convincingly feign ignorance. We should make tools readily available and user friendly so it is easier for people to detect CSAM that they have unintentionally interacted with. This both shields the innocent from being falsely accused, and makes it easier to stop bad actors as their activities are detected earlier.
    - pixl97 10mo ago
      No, it should be law enforcement job to determine intent, not a blanket you're guilty. This being Actus Reus is a huge mess that makes it easy to frame people and get in trouble with no guilty act.
      - jjk166 10mo ago
        Determining intent takes time, is often not possible, and encourages people to specifically avoid the work to check if something needs to be flagged. Not checking is at best negligent. Having everybody check and flag is the sensible option.
        pixl97 10mo ago
        Ah, yes, the assume everyone is guilty and let god sort them out method. Authoritarians love it.
        jjk166 10mo ago
        Everyone in this case meaning "people demonstrated to be in possession of child porn who took no action". And they are not assumed guilty, they are exactly as innocent as anyone with a dead body in their fridge that they also "had no idea about."
  - burnt-resistor 10mo ago
    That's the root problem with all mandated, invasive CSAM scanning. (Non-signature based) creates an unreasonable panopticon that leads to lifelong banishment by imprecise, evidence-free guessing. It also hyper-criminalizes every parent who accidentally takes a picture of their kid without being fully dressed. And what about DoS victims who are anonymously sent CSAM without their consent to get them banned for "possession"? While pedo is gross and evil no doubt, but extreme "think of the children" measures that sacrifice liberty and privacy create another evil that is different. Handing over total responsibility and ultimate decision-making for critical matters to a flawed algorithm is lazy, negligent, and immoral. There's no easy solution to any such process, except requiring human review should be the moral and ethical minimum standard before drastic measures (human in the loop (HITL)).
- jmogly 10mo ago
  On one hand, I would like to say this could happen to anyone, on the other hand, what the F?? why are people passing around a dataset that contains child sexual abuse material??, and on another hand, I think this whole thing just reeks of techy-bravado, and I don’t exactly blame him. If one of the inputs of your product (OpenAI, google, microsoft, meta, X) is a dataset that you can’t even say for sure does not contain child pornography, that’s pretty alarming.
deltoidmaximus 10mo ago
Back when the first moat creation gambit for AI failed (that they were creating SkyNet so the government needs to block anyone else from working on SkyNet since only OpenAI can be trusted to control it not just any rando) they moved onto the safety angle with the same idea. I recall seeing an infographic that all the major players were signed onto some kind of safety pledge, Meta, OpenAI, Microsoft, etc. Basically they didn't want anyone else training on the whole world's data because only they could be trusted to not do nefarious things with it. The infographic had a statement about not training on CSAM and revenge porn and the like but the corpospeak it was worded in made it sound like they were promising not to do it anymore, not that they never did. I've tried to find this graphic against several times over the years but it's either been scrubbed from the internet or I just can't remember enough details to find it. Amusingly, it only just occurred to me that maybe I should ask ChatGPT to help me find it.
- jsheard 10mo ago
  > The infographic had a statement about not training on CSAM and revenge porn and the like but the corpospeak it was worded in made it sound like they were promising not to do it anymore, not that they never did. We know they did, an earlier version of the LAION dataset was found to contain CSAM after everyone had already trained their image generation models on it. https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford https://www.theverge.com/2023/12/20/24009418/generative-ai-i...
jsnell 10mo ago
As a small point of order, they did not get banned for "finding CSAM" like the outrage- and clickbait title claims. They got banned for uploading a data set containing child porn to Google Drive. They did not find it themselves, and them later reporting the data set to an appropriate organization is not why they got banned.
- jeffbee 10mo ago
  Literally every headline that 404 media has published about subjects I understand first-hand has been false.
  - amelius 10mo ago
    Can we use AI to fix this? Make an LLM read the articles behind the links, and then rewrite the headlines (in a browser plugin for instance).
    - add-sub-mul-div 10mo ago
      HN already needlessly rewrites headlines with automation and it's more annoying to see automation go stupidly wrong than letting the original imperfect situation stand. Having outrage about headlines is a choice.
      - amelius 10mo ago
        I don't think HN's rewrite algorithm uses modern LLM techniques. Also, it could be optional. It probably should be, in fact.
        jeffbee 10mo ago
        My browser integrates an LLM, so I asked it to restate the headline of this one, and it came up with "Developer Suspended by Google After Uploading AI Dataset Containing CSAM" which seems pretty even-handed. Of course, I would want to dial the snark to 11. Many hacker stories can be headlined "Developer discovers that C still sucks" etc.
giantg2 10mo ago
This raises an interesting point. Do you need to train models using CSAM so that the model can self-enforce restrictions on CSAM? If so, I wonder what moral/ethical questions this brings up.
- jsheard 10mo ago
  It's a delicate subject but not an unprecedented one. Automatic detection of already known CSAM images (as opposed to heuristic detection of unknown images) has been around for much longer than AI, and for that service to exist someone has to handle the actual CSAM before it's reduced to a perceptual hash in a database. Maybe AI-based heuristic detection is more ethically/legally fraught since you'd have to stockpile CSAM to train on, rather than hashing then destroying your copy immediately after obtaining it.
  - tcfhgj 10mo ago
    > Maybe AI detection is more ethically fraught since you'd need to keep hold of the CSAM until the next training run, why? the damage is already done
    - tremon 10mo ago
      Why would you think that? Every distribution, every view is adding damage, even if the original victim doesn't know (or even would rather not know) about it.
      - jjk166 10mo ago
        I don't think AI training on a dataset counts as a view in this context. The concern is predators getting off on what they've done, not developing tools to stop them.
        pseudalopex 10mo ago
        Debating what counts as a view is irrelevant. Some child pornography subjects feel violated by any storage or use of their images. Government officials store and use them regardless.
amarcheschi 10mo ago
Just a few days ago I was doing some low paid (well, not so low) Ai classification task - akin to mechanical turk ones - for a very big company and was - involuntarily, since I guess they don't review them before showing - shown an ai image by the platform depicting a naked man and naked kid. though it was more barbie like than anything else. I didn't really enjoy the view tbh, contacted them but got no answer back
- ipython 10mo ago
  If the picture truly was of a child, the company is _required_ to report CSAM to NCMEC. It's taken very seriously. If they're not being responsive, escalate and report it yourself so you don't have legal problems. See https://report.cybertip.org/ https://report.cybertip.org/.
  - amarcheschi 10mo ago
    Even if it's an Ai image? I will follow through contacting them directly rather than with the platform messaging system, then I'll see what to do if they don't answer Edit i read the informations given in the briefing before the task, and they say that there might be offensive content displayed. They say to tell them if it happens, but well I did and got no answer so weeeell, not so inclined to believe they care about it
    - jfindper 10mo ago
      >Even if it's an Ai image? This varies by country, but in many countries it doesn't matter if it is a drawing, AI, or a real image -- they are treated equally for the purposes of CSAM.
      - amarcheschi 10mo ago
        That's understandable
    - ipython 10mo ago
      The company may not care, but the gov definitely does. And if you don’t report then you could be in serious legal jeopardy. If any fragments of that image are still present on your machine, whether it came from the company or not, you could be held accountable for possessing csam. So screw the company, report it yourself and make sure to cite the company and their lack of a response. There’s a Grand Canyon sized chasm between “offensive content” and csam.
gillesjacobs 10mo ago
https://archive.ph/awvmJ https://archive.ph/awvmJ
winchester6788 10mo ago
Author of NudeNet here. I just scraped data from reddit and other sources so i could build a nsfw classifier and chose to open source the data and the model for general good. Note that i was a 1 year experienced engineer working solely on this project in my free time, so it was basically impossible for me to review or clear out the few csam images in the 100,000+ images in the dataset. Although, now i wonder if i should never have open sourced the data. Would have avoided lot of these issues.
- markatlarge 10mo ago
  Im the developers who actually got banned because of this dataset. I used NudeNet offline to benchmark my on-device NSFW app Punge — nothing uploaded, nothing shared. Your dataset wasn’t the problem. The real problem is that independent developers have zero access to the tools needed to detect CSAM, while Big Tech keeps those capabilities to itself. Meanwhile, Google and other giants openly use massive datasets like LAION-5B — which also contained CSAM — without facing any consequences at all. Google even used early LAION data to train one of its own models. Nobody bans Google. But when I touched NudeNet for legitimate testing, Google deleted 130,000+ files from my account, even though only ~700 images out of ~700,000 were actually problematic. That’s not safety — that’s a detection system wildly over firing with no independent oversight and no accountability. Big Tech designed a world where they alone have the scanning tools and the immunity when those tools fail. Everyone else gets punished for their mistakes. So yes — your dataset has done good. ANY data set is subject to this. There needs to be tools and process for all. But let’s be honest about where the harm came from: a system rigged so only Big Tech can safely build or host datasets, while indie developers get wiped out by the exact same automated systems Big Tech exempts itself from.
  - petee 10mo ago
    700 were csam, if I'm reading this right?
    - wang_li 10mo ago
      Perhaps these folks should work together to make patches to the dataset to remove the problematic images? E: But also make sure every image in the dataset is properly licensed. This would have eliminated this entirely from the get go. Playing fast and loose with the distribution rights to these images led to this problem.
UberFly 10mo ago
Posting articles that are paywalled is worthless.
- deleted 10mo ago
  [deleted]
- stronglikedan 10mo ago
  Finding the non-paywalled link in the comments is trivial.
mflkgknr 10mo ago
Being banned for pointing out ”Emperor’s new clothes”, is what autocrats typically do, because the worst thing they know is when anyone embarrass them.
codedokode 10mo ago
Slightly unrelated, but I wonder if a 17-year old child sends her dirty photo to a 18-year old guy she likes, who goes to jail? Just curious how the law works if there is no "abuse" element.
- jfindper 10mo ago
  Obviously this depends on the country, but many countries have so-called "Romeo and Juliet" laws which carve out specific exclusions for situations along these lines.
  - stavros 10mo ago
    Those tend to be about sex, not pornography, no?
    - jfindper 10mo ago
      I'm certainly no expert, but if my memory serves me correctly from cases that have hit the media, the carve outs are more broadly applicable than just to sex (e.g. intimate images between partners). But I could certainly be wrong! (I didn't really want to start looking up the exact details of this topic while at work, so just went from memory. At the very least, the terminology "Romeo & Juliet Law" should give the original commenter enough to base a search on)
- xethos 10mo ago
  One for distribution (of her own image), one for possession. See sections 3 & 4 https://laws-lois.justice.gc.ca/eng/acts/c-46/section-163.1.html https://laws-lois.justice.gc.ca/eng/acts/c-46/section-163.1....
- bigfatkitten 10mo ago
  It comes down to prosecutorial discretion, and that can go either way. Prosecutors have broad discretion to proceed with a matter based on whether there is a reasonable prospect of securing a conviction, whether it’s in the public interest to do so and various other factors. They don’t generally bring a lot of rigour to these considerations.
burnt-resistor 10mo ago
Technofeudalism strikes again. MAANG can ban people at any time for anything without appeal, and sometimes at the whim of any nation state. Reversal is the rare exception, not the rule, and only happens occasionally due to public pressure.
josefritzishere 10mo ago
Oh gross

122 points by markatlarge 10mo ago

bsowl 10mo ago

More like "A developer accidentally uploaded child porn to his Google Drive account and Google banned him for it".

jkaplowitz 10mo ago
The penalties for unknowingly possessing or transmitting child porn are far too harsh, both in this case and in general (far beyond just Google's corporate policies). Again, to avoid misunderstandings, I said unknowingly - I'm not defending anything about people who knowingly possess or traffic in child porn, other than for the few appropriate purposes like reporting it to the proper authorities when discovered.
- jjk166 10mo ago
  The issue is that when you make ignorance a valid defense, the optimal strategy is to deliberately turn a blind eye, as it reduces your risk exposure. It further gives refuge for those who can convincingly feign ignorance. We should make tools readily available and user friendly so it is easier for people to detect CSAM that they have unintentionally interacted with. This both shields the innocent from being falsely accused, and makes it easier to stop bad actors as their activities are detected earlier.
  - pixl97 10mo ago
    No, it should be law enforcement job to determine intent, not a blanket you're guilty. This being Actus Reus is a huge mess that makes it easy to frame people and get in trouble with no guilty act.
    - jjk166 10mo ago
      Determining intent takes time, is often not possible, and encourages people to specifically avoid the work to check if something needs to be flagged. Not checking is at best negligent. Having everybody check and flag is the sensible option.
      - pixl97 10mo ago
        Ah, yes, the assume everyone is guilty and let god sort them out method. Authoritarians love it.
        jjk166 10mo ago
        Everyone in this case meaning "people demonstrated to be in possession of child porn who took no action". And they are not assumed guilty, they are exactly as innocent as anyone with a dead body in their fridge that they also "had no idea about."
- burnt-resistor 10mo ago
  That's the root problem with all mandated, invasive CSAM scanning. (Non-signature based) creates an unreasonable panopticon that leads to lifelong banishment by imprecise, evidence-free guessing. It also hyper-criminalizes every parent who accidentally takes a picture of their kid without being fully dressed. And what about DoS victims who are anonymously sent CSAM without their consent to get them banned for "possession"? While pedo is gross and evil no doubt, but extreme "think of the children" measures that sacrifice liberty and privacy create another evil that is different. Handing over total responsibility and ultimate decision-making for critical matters to a flawed algorithm is lazy, negligent, and immoral. There's no easy solution to any such process, except requiring human review should be the moral and ethical minimum standard before drastic measures (human in the loop (HITL)).
jmogly 10mo ago
On one hand, I would like to say this could happen to anyone, on the other hand, what the F?? why are people passing around a dataset that contains child sexual abuse material??, and on another hand, I think this whole thing just reeks of techy-bravado, and I don’t exactly blame him. If one of the inputs of your product (OpenAI, google, microsoft, meta, X) is a dataset that you can’t even say for sure does not contain child pornography, that’s pretty alarming.

deltoidmaximus 10mo ago

Back when the first moat creation gambit for AI failed (that they were creating SkyNet so the government needs to block anyone else from working on SkyNet since only OpenAI can be trusted to control it not just any rando) they moved onto the safety angle with the same idea. I recall seeing an infographic that all the major players were signed onto some kind of safety pledge, Meta, OpenAI, Microsoft, etc. Basically they didn't want anyone else training on the whole world's data because only they could be trusted to not do nefarious things with it. The infographic had a statement about not training on CSAM and revenge porn and the like but the corpospeak it was worded in made it sound like they were promising not to do it anymore, not that they never did. I've tried to find this graphic against several times over the years but it's either been scrubbed from the internet or I just can't remember enough details to find it. Amusingly, it only just occurred to me that maybe I should ask ChatGPT to help me find it.

jsheard 10mo ago
> The infographic had a statement about not training on CSAM and revenge porn and the like but the corpospeak it was worded in made it sound like they were promising not to do it anymore, not that they never did. We know they did, an earlier version of the LAION dataset was found to contain CSAM after everyone had already trained their image generation models on it. https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford https://www.theverge.com/2023/12/20/24009418/generative-ai-i...

jsnell 10mo ago

As a small point of order, they did not get banned for "finding CSAM" like the outrage- and clickbait title claims. They got banned for uploading a data set containing child porn to Google Drive. They did not find it themselves, and them later reporting the data set to an appropriate organization is not why they got banned.

jeffbee 10mo ago
Literally every headline that 404 media has published about subjects I understand first-hand has been false.
- amelius 10mo ago
  Can we use AI to fix this? Make an LLM read the articles behind the links, and then rewrite the headlines (in a browser plugin for instance).
  - add-sub-mul-div 10mo ago
    HN already needlessly rewrites headlines with automation and it's more annoying to see automation go stupidly wrong than letting the original imperfect situation stand. Having outrage about headlines is a choice.
    - amelius 10mo ago
      I don't think HN's rewrite algorithm uses modern LLM techniques. Also, it could be optional. It probably should be, in fact.
      - jeffbee 10mo ago
        My browser integrates an LLM, so I asked it to restate the headline of this one, and it came up with "Developer Suspended by Google After Uploading AI Dataset Containing CSAM" which seems pretty even-handed. Of course, I would want to dial the snark to 11. Many hacker stories can be headlined "Developer discovers that C still sucks" etc.

giantg2 10mo ago

This raises an interesting point. Do you need to train models using CSAM so that the model can self-enforce restrictions on CSAM? If so, I wonder what moral/ethical questions this brings up.

jsheard 10mo ago
It's a delicate subject but not an unprecedented one. Automatic detection of already known CSAM images (as opposed to heuristic detection of unknown images) has been around for much longer than AI, and for that service to exist someone has to handle the actual CSAM before it's reduced to a perceptual hash in a database. Maybe AI-based heuristic detection is more ethically/legally fraught since you'd have to stockpile CSAM to train on, rather than hashing then destroying your copy immediately after obtaining it.
- tcfhgj 10mo ago
  > Maybe AI detection is more ethically fraught since you'd need to keep hold of the CSAM until the next training run, why? the damage is already done
  - tremon 10mo ago
    Why would you think that? Every distribution, every view is adding damage, even if the original victim doesn't know (or even would rather not know) about it.
    - jjk166 10mo ago
      I don't think AI training on a dataset counts as a view in this context. The concern is predators getting off on what they've done, not developing tools to stop them.
      - pseudalopex 10mo ago
        Debating what counts as a view is irrelevant. Some child pornography subjects feel violated by any storage or use of their images. Government officials store and use them regardless.

amarcheschi 10mo ago

Just a few days ago I was doing some low paid (well, not so low) Ai classification task - akin to mechanical turk ones - for a very big company and was - involuntarily, since I guess they don't review them before showing - shown an ai image by the platform depicting a naked man and naked kid. though it was more barbie like than anything else. I didn't really enjoy the view tbh, contacted them but got no answer back

ipython 10mo ago
If the picture truly was of a child, the company is _required_ to report CSAM to NCMEC. It's taken very seriously. If they're not being responsive, escalate and report it yourself so you don't have legal problems. See https://report.cybertip.org/ https://report.cybertip.org/.
- amarcheschi 10mo ago
  Even if it's an Ai image? I will follow through contacting them directly rather than with the platform messaging system, then I'll see what to do if they don't answer Edit i read the informations given in the briefing before the task, and they say that there might be offensive content displayed. They say to tell them if it happens, but well I did and got no answer so weeeell, not so inclined to believe they care about it
  - jfindper 10mo ago
    >Even if it's an Ai image? This varies by country, but in many countries it doesn't matter if it is a drawing, AI, or a real image -- they are treated equally for the purposes of CSAM.
    - amarcheschi 10mo ago
      That's understandable
  - ipython 10mo ago
    The company may not care, but the gov definitely does. And if you don’t report then you could be in serious legal jeopardy. If any fragments of that image are still present on your machine, whether it came from the company or not, you could be held accountable for possessing csam. So screw the company, report it yourself and make sure to cite the company and their lack of a response. There’s a Grand Canyon sized chasm between “offensive content” and csam.

gillesjacobs 10mo ago

https://archive.ph/awvmJ https://archive.ph/awvmJ

winchester6788 10mo ago

Author of NudeNet here. I just scraped data from reddit and other sources so i could build a nsfw classifier and chose to open source the data and the model for general good. Note that i was a 1 year experienced engineer working solely on this project in my free time, so it was basically impossible for me to review or clear out the few csam images in the 100,000+ images in the dataset. Although, now i wonder if i should never have open sourced the data. Would have avoided lot of these issues.

markatlarge 10mo ago
Im the developers who actually got banned because of this dataset. I used NudeNet offline to benchmark my on-device NSFW app Punge — nothing uploaded, nothing shared. Your dataset wasn’t the problem. The real problem is that independent developers have zero access to the tools needed to detect CSAM, while Big Tech keeps those capabilities to itself. Meanwhile, Google and other giants openly use massive datasets like LAION-5B — which also contained CSAM — without facing any consequences at all. Google even used early LAION data to train one of its own models. Nobody bans Google. But when I touched NudeNet for legitimate testing, Google deleted 130,000+ files from my account, even though only ~700 images out of ~700,000 were actually problematic. That’s not safety — that’s a detection system wildly over firing with no independent oversight and no accountability. Big Tech designed a world where they alone have the scanning tools and the immunity when those tools fail. Everyone else gets punished for their mistakes. So yes — your dataset has done good. ANY data set is subject to this. There needs to be tools and process for all. But let’s be honest about where the harm came from: a system rigged so only Big Tech can safely build or host datasets, while indie developers get wiped out by the exact same automated systems Big Tech exempts itself from.
- petee 10mo ago
  700 were csam, if I'm reading this right?
  - wang_li 10mo ago
    Perhaps these folks should work together to make patches to the dataset to remove the problematic images? E: But also make sure every image in the dataset is properly licensed. This would have eliminated this entirely from the get go. Playing fast and loose with the distribution rights to these images led to this problem.

UberFly 10mo ago

Posting articles that are paywalled is worthless.

deleted 10mo ago
[deleted]
stronglikedan 10mo ago
Finding the non-paywalled link in the comments is trivial.

mflkgknr 10mo ago

Being banned for pointing out ”Emperor’s new clothes”, is what autocrats typically do, because the worst thing they know is when anyone embarrass them.

codedokode 10mo ago

Slightly unrelated, but I wonder if a 17-year old child sends her dirty photo to a 18-year old guy she likes, who goes to jail? Just curious how the law works if there is no "abuse" element.

jfindper 10mo ago
Obviously this depends on the country, but many countries have so-called "Romeo and Juliet" laws which carve out specific exclusions for situations along these lines.
- stavros 10mo ago
  Those tend to be about sex, not pornography, no?
  - jfindper 10mo ago
    I'm certainly no expert, but if my memory serves me correctly from cases that have hit the media, the carve outs are more broadly applicable than just to sex (e.g. intimate images between partners). But I could certainly be wrong! (I didn't really want to start looking up the exact details of this topic while at work, so just went from memory. At the very least, the terminology "Romeo & Juliet Law" should give the original commenter enough to base a search on)
xethos 10mo ago
One for distribution (of her own image), one for possession. See sections 3 & 4 https://laws-lois.justice.gc.ca/eng/acts/c-46/section-163.1.html https://laws-lois.justice.gc.ca/eng/acts/c-46/section-163.1....
bigfatkitten 10mo ago
It comes down to prosecutorial discretion, and that can go either way. Prosecutors have broad discretion to proceed with a matter based on whether there is a reasonable prospect of securing a conviction, whether it’s in the public interest to do so and various other factors. They don’t generally bring a lot of rigour to these considerations.

burnt-resistor 10mo ago

Technofeudalism strikes again. MAANG can ban people at any time for anything without appeal, and sometimes at the whim of any nation state. Reversal is the rare exception, not the rule, and only happens occasionally due to public pressure.

josefritzishere 10mo ago

Oh gross