5 ms·

I guess they had a system that required humans to confirm the validity or so. I had a similar problem when I realized someone put up CSAM (Children sexual abus

I guess they had a system that required humans to confirm the validity or so. I had a similar problem when I realized someone put up CSAM (Children sexual abuse material) on my public demo of my CDN called PictShare [1]. I didn't want to look through all of these images so I built a Raspberry Pi with a Neural compute stick [2] that used an AI model trained by yahoo to filter out "nudity" images and I put them in an encrypted ZIP file along with the access logs and sent them to Interpol. This lead to the arrest of a teacher here in Austria so I'm glad I could do my part. This even lead to a BBC article about my system [3] [1] https://github.com/HaschekSolutions/pictshare https://github.com/HaschekSolutions/pictshare [2] https://blog.haschek.at/2018/fight-child-pornography-with-raspi-and-deep-learning.html https://blog.haschek.at/2018/fight-child-pornography-with-ra... [3] https://www.bbc.com/news/technology-44525358 https://www.bbc.com/news/technology-44525358

throwaway290 3y ago
This is awesome, but since you mention you used a nudity detector and haven't reviewed them... I would expect for a popular service you would have gigabytes of random legal nudity for every actual Interpol-worthy image, right? I guess Interpol is OK sifting through all that? And wouldn't that mean that for all of those nude-but-legal pics you were unnecessarily disclosing regular user PII by sending along access logs to law enforcement? Not to say it wasn't worth it in the end
- geek_at 3y ago
  I asked interpol upfront if it would be okay to send them the data for them to review and they said OK. There would probably not be a reason to upload "legal nudity" on a demo site of a selfhostable service and also I wipe the whole database in irregular intervals. But actually now I'm using the CSAM detection tool from cloudflare which automatically detects, flags and reports CSAM which takes the hassle out for me. Not sure about false positives though but afterall. It's just a demo site and nobody should upload anything other than test things
- selcuka 3y ago
  Note that they said "I didn't want to look through all of these images", not the ones that are detected. They probably manually reviewed the ones that are detected by the automated system before sending them to Interpol.
- isaacfrond 3y ago
  The more detailed stories in the link do not mention that the material was forwarded unreviewed. In fact, they specifically mention that this concerns 16 images.
  - throwaway290 3y ago
    To me it did (and does) sound like service operator did a very straightforward "detect nudity -> zip -> Interpol" at first.
Semaphor 3y ago
Good thing you aren’t German where this would now probably be illegal because we have politicians that are scared of thinking (possession of CSAM leaves no more way for a judge or a prosecutor to not prosecute or stopping, no matter the reason, even if the reason is to bring it to the police, a proud tradition in Germany, making the removal of CSAM harder, that Ursula von der Leyen started). A German legal discussion about this is here [0] where a teacher who wanted to help is now being prosecuted (simpler summary here [1]), and apparently those idiots now finally plan to change the law after exactly those issues were explained to them when they started implementing this law. [0]: https://community.beck.de/2023/08/30/wenn-gesetzgeber-und-justizminister-in-fehler-machen-dann-gibt-es-kein-pardon-besitz-von-kinderpornographie https://community.beck.de/2023/08/30/wenn-gesetzgeber-und-ju... [1]: https://www.lawblog.de/archives/2023/08/28/staatsanwaltschaft-bedauert-dass-sie-lehrerin-anklagen-muss/ https://www.lawblog.de/archives/2023/08/28/staatsanwaltschaf...
- geek_at 3y ago
  Oh wow that's really fucked up. Here in Austria the police is not very well trained on any computer related crimes either. When I initially found the first CSAM image my first instinct was to call the non-emergency line of the Police and the officer told me to "print out the images, and bring them to the next police station". I had a thought that this would be a very very stupid thing to do so I contacted Interpol and they sad basically "omg whatever you do, don't print those out and don't carry them around"
  - Semaphor 3y ago
    > "print out the images, and bring them to the next police station". Wow, I think that would have been an issue here even before the change, only then the prosecutor could have decided to stop the proceedings before it even went to court.
  - lynx23 3y ago
    Ahh, austria police. A few years ago, my neighbour had an aggressive episode and ended up telling me "I bring di um". Naiv as I was, I called the police, which promptly answered "Do kemma nix tuan".

throwaway290 3y ago

This is awesome, but since you mention you used a nudity detector and haven't reviewed them... I would expect for a popular service you would have gigabytes of random legal nudity for every actual Interpol-worthy image, right? I guess Interpol is OK sifting through all that? And wouldn't that mean that for all of those nude-but-legal pics you were unnecessarily disclosing regular user PII by sending along access logs to law enforcement? Not to say it wasn't worth it in the end

geek_at 3y ago
I asked interpol upfront if it would be okay to send them the data for them to review and they said OK. There would probably not be a reason to upload "legal nudity" on a demo site of a selfhostable service and also I wipe the whole database in irregular intervals. But actually now I'm using the CSAM detection tool from cloudflare which automatically detects, flags and reports CSAM which takes the hassle out for me. Not sure about false positives though but afterall. It's just a demo site and nobody should upload anything other than test things
selcuka 3y ago
Note that they said "I didn't want to look through all of these images", not the ones that are detected. They probably manually reviewed the ones that are detected by the automated system before sending them to Interpol.
isaacfrond 3y ago
The more detailed stories in the link do not mention that the material was forwarded unreviewed. In fact, they specifically mention that this concerns 16 images.
- throwaway290 3y ago
  To me it did (and does) sound like service operator did a very straightforward "detect nudity -> zip -> Interpol" at first.

Semaphor 3y ago

Good thing you aren’t German where this would now probably be illegal because we have politicians that are scared of thinking (possession of CSAM leaves no more way for a judge or a prosecutor to not prosecute or stopping, no matter the reason, even if the reason is to bring it to the police, a proud tradition in Germany, making the removal of CSAM harder, that Ursula von der Leyen started). A German legal discussion about this is here [0] where a teacher who wanted to help is now being prosecuted (simpler summary here [1]), and apparently those idiots now finally plan to change the law after exactly those issues were explained to them when they started implementing this law. [0]: https://community.beck.de/2023/08/30/wenn-gesetzgeber-und-justizminister-in-fehler-machen-dann-gibt-es-kein-pardon-besitz-von-kinderpornographie https://community.beck.de/2023/08/30/wenn-gesetzgeber-und-ju... [1]: https://www.lawblog.de/archives/2023/08/28/staatsanwaltschaft-bedauert-dass-sie-lehrerin-anklagen-muss/ https://www.lawblog.de/archives/2023/08/28/staatsanwaltschaf...

geek_at 3y ago
Oh wow that's really fucked up. Here in Austria the police is not very well trained on any computer related crimes either. When I initially found the first CSAM image my first instinct was to call the non-emergency line of the Police and the officer told me to "print out the images, and bring them to the next police station". I had a thought that this would be a very very stupid thing to do so I contacted Interpol and they sad basically "omg whatever you do, don't print those out and don't carry them around"
- Semaphor 3y ago
  > "print out the images, and bring them to the next police station". Wow, I think that would have been an issue here even before the change, only then the prosecutor could have decided to stop the proceedings before it even went to court.
- lynx23 3y ago
  Ahh, austria police. A few years ago, my neighbour had an aggressive episode and ended up telling me "I bring di um". Naiv as I was, I called the police, which promptly answered "Do kemma nix tuan".