4 ms·

If governments and big tech want to help, they should upload one of their CSAM detection models to Hugging Face, so system administrators can just block it. Ide

by jart 2y ago

If governments and big tech want to help, they should upload one of their CSAM detection models to Hugging Face, so system administrators can just block it. Ideally I should be able to run a command `iscsam 123.jpg` and it prints a number like 0.9 to indicate 90% confidence that it is. No one else but them can do it, since there's obviously no legal way to train such a model. Even though we know that governments have already done it. If they won't give service operators the tools to keep abuse off their communications systems, then operators shouldn't be held accountable for what people do with them.

kevindamm 2y ago
The biggest risk with opening a tool like that is that it potentially enables offenders to figure out what can get past it.
- jart 2y ago
  So they publish an updated model every three months that works better.
- marpstar 2y ago
  Fair point, but wouldn’t we rather they be spending their time doing that than actively abusing kids?
- cr125rider 2y ago
  So security by obscurity? Man, open source software must suck…
blackoil 2y ago
Perpetrators will keep tweaking image till they get score of 0.1
- amelius 2y ago
  How about the government running a service where you can ask them to validate an image? Trying to tweak an image will not work because you will find the police on your doorstep.
  - jart 2y ago
    The government doesn't need more dragnet surveillance capabilities than it already has. Also this solution basically asks the service operator to upload illegal content to the government. So there would need to be a strong guarantee they wouldn't use that as proof the service operator has committed a crime. Imagine what they would do to Elon Musk if he did that to run X. The government is also usually incompetent at running reliable services.
    - bigfudge 2y ago
      "The government" in the UK already basically shields big internet operators from legal responsibility from showing teenagers how to tie a ligature. But I wouldn't characterise them as the problem — more public oversight or at least transparency of the behaviour of online operators who run services used by thousands of minors might not be a bad thing. The Musk comment also speaks to a paranoia that just isn't justified by anything that has happened in the past 10 years. The EU is in fact the only governmental organisation doing anything to constrain the power of billionaires to distort and control our public discourse through mass media and social media ownership.
      - jart 2y ago
        You mean the government of Prince Andrew? Yeah I think I understand now why they want the csam so badly.
        bigfudge 2y ago
        I don't understand this comment. Are you implying Prince Andrew was _in_ or part of the UK Government? This would be a weird misunderstanding of our system. If it's just a general cynical "all gubernment is bad and full of pedos" then I'm not sure what the comment adds to this discussion.
  - charrondev 2y ago
    My understanding is at that Microsoft runs such a tool and you can request access to it. (PhotoDNA). As I understand you hash an image send it to them and get back a response.
- baby_souffle 2y ago
  > Perpetrators will keep tweaking image till they get score of 0.1 Isn't this - more or less - already happening? Perpetrators that don't find _some way_ of creating/sharing csam that's low risk get arrested. The "fear of being in jail" is already driving these people to invent/seek out ways to score a 0.1.
miki123211 2y ago
This would potentially let somebody create a "reverse" model, so I don't think that's a good idea. Imagine an image generation model whose loss function is essentially "make this other model classify your image as CSAM." I'm not entirely convinced whether it would create actual CSAM instead of adversarial examples, but we've seen other models of various kinds "reversed" in a similar vein, so I think there's quite a bit of risk there.
- jart 2y ago
  Are you saying someone will use it to create a CSAM generator? It'd be like turning smoke detectors into a nuclear bomb. If someone that smart wants this, then there are easier ways for them to do it. Analyzing the detector could let you tune normal images in an adversarial way that'll cause them to be detected as CSAM by a specific release of a specific model. So long as you're not using the model to automate swatting, that's not going to amount to much more than a DEFCON talk about annoying people.
  - throwaway290 2y ago
    I think the point is generating an image that looks normal but causes the model to false positive and the unsuspecting person then gets reported
    - jart 2y ago
      If you have a csam detection model that can run locally, the vast majority of sysadmins who use it will just delete the content and ban whoever posted it. Why would they report someone to the police? If you're running a file sharing service, you probably don't even know the identities of your users. You could try looking up the user IP on WHOIS and emailing the abuse contact, but chances are no one is listening and no one will care. What's important is that (1) it'll be harder to distribute this material, (2) service operators who are just trying to build and innovate will be able to easily protect themselves with minimal cost.
      - 2y ago
tonetegeatinst 2y ago
Pretty sure apple already scans your photos for csam, so the best way would be to just throw any files a user plans on sharing into some folder an iPhone or iMac has access to.

by jart 2y ago

kevindamm 2y ago

The biggest risk with opening a tool like that is that it potentially enables offenders to figure out what can get past it.

jart 2y ago
So they publish an updated model every three months that works better.
marpstar 2y ago
Fair point, but wouldn’t we rather they be spending their time doing that than actively abusing kids?
cr125rider 2y ago
So security by obscurity? Man, open source software must suck…

blackoil 2y ago

Perpetrators will keep tweaking image till they get score of 0.1

amelius 2y ago
How about the government running a service where you can ask them to validate an image? Trying to tweak an image will not work because you will find the police on your doorstep.
- jart 2y ago
  The government doesn't need more dragnet surveillance capabilities than it already has. Also this solution basically asks the service operator to upload illegal content to the government. So there would need to be a strong guarantee they wouldn't use that as proof the service operator has committed a crime. Imagine what they would do to Elon Musk if he did that to run X. The government is also usually incompetent at running reliable services.
  - bigfudge 2y ago
    "The government" in the UK already basically shields big internet operators from legal responsibility from showing teenagers how to tie a ligature. But I wouldn't characterise them as the problem — more public oversight or at least transparency of the behaviour of online operators who run services used by thousands of minors might not be a bad thing. The Musk comment also speaks to a paranoia that just isn't justified by anything that has happened in the past 10 years. The EU is in fact the only governmental organisation doing anything to constrain the power of billionaires to distort and control our public discourse through mass media and social media ownership.
    - jart 2y ago
      You mean the government of Prince Andrew? Yeah I think I understand now why they want the csam so badly.
      - bigfudge 2y ago
        I don't understand this comment. Are you implying Prince Andrew was _in_ or part of the UK Government? This would be a weird misunderstanding of our system. If it's just a general cynical "all gubernment is bad and full of pedos" then I'm not sure what the comment adds to this discussion.
- charrondev 2y ago
  My understanding is at that Microsoft runs such a tool and you can request access to it. (PhotoDNA). As I understand you hash an image send it to them and get back a response.
baby_souffle 2y ago
> Perpetrators will keep tweaking image till they get score of 0.1 Isn't this - more or less - already happening? Perpetrators that don't find _some way_ of creating/sharing csam that's low risk get arrested. The "fear of being in jail" is already driving these people to invent/seek out ways to score a 0.1.

miki123211 2y ago

This would potentially let somebody create a "reverse" model, so I don't think that's a good idea. Imagine an image generation model whose loss function is essentially "make this other model classify your image as CSAM." I'm not entirely convinced whether it would create actual CSAM instead of adversarial examples, but we've seen other models of various kinds "reversed" in a similar vein, so I think there's quite a bit of risk there.

jart 2y ago
Are you saying someone will use it to create a CSAM generator? It'd be like turning smoke detectors into a nuclear bomb. If someone that smart wants this, then there are easier ways for them to do it. Analyzing the detector could let you tune normal images in an adversarial way that'll cause them to be detected as CSAM by a specific release of a specific model. So long as you're not using the model to automate swatting, that's not going to amount to much more than a DEFCON talk about annoying people.
- throwaway290 2y ago
  I think the point is generating an image that looks normal but causes the model to false positive and the unsuspecting person then gets reported
  - jart 2y ago
    If you have a csam detection model that can run locally, the vast majority of sysadmins who use it will just delete the content and ban whoever posted it. Why would they report someone to the police? If you're running a file sharing service, you probably don't even know the identities of your users. You could try looking up the user IP on WHOIS and emailing the abuse contact, but chances are no one is listening and no one will care. What's important is that (1) it'll be harder to distribute this material, (2) service operators who are just trying to build and innovate will be able to easily protect themselves with minimal cost.
    - 2y ago

tonetegeatinst 2y ago

Pretty sure apple already scans your photos for csam, so the best way would be to just throw any files a user plans on sharing into some folder an iPhone or iMac has access to.