5 ms·

There is: PhotoDNA. The name sounds neutral, by design, but don’t let it fool you: unless your boss explicitly tells you this is your project now, I’d recommend

by bertil 8y ago

There is: PhotoDNA. The name sounds neutral, by design, but don’t let it fool you: unless your boss explicitly tells you this is your project now, I’d recommend you stay far away. Well, I’m not sure it’s FOSS or anything like that: it’s maintained by the police (a weak assortment of the services that handle that in different countries) and it has a key feature: it’s a bloom/hash filter of sorts, i.e. you don’t need to open the image to make it work (because even opening the file would be illegal). It recognises photos similar to those already flagged (which are absolutely not free, open or available in any way for obvious reasons). A typical implementation is to move the flagged image in a dedicated folder with very strict access-control, keep really reliable traceability and send details to your local dedicated police team. Even access to the filters is tightly controlled too, because the wrong kind of people could use it to sift through a lot of images. Because it’s a fairly standard tool for any image hosting company, I’m surprised Tumblr has been shamed. Either they messed up their integration, or they ignored police requests. It’s odd.

ballenf 8y ago
> even opening the file would be illegal Not an expert, but I've worked with large companies that used humans to filter user content. Either they were breaking the law en masse deliberately or there is a safe harbor for doing manual filtering. Again, I don't know the answer, but I'm very skeptical that it's always illegal to manually filter user content. Presumably swift deletion (and cache purging, if possible), reporting and record-keeping would be important. Although that does raise the issue of how you preserve evidence for police to take action if you delete it immediately... geez this is such a landmine.
- MrStonedOne 8y ago
  There is an affirmative defence in most jurisdictions for reporting cp you come in contact with.
- bertil 8y ago
  Not an expert either, but I believe that once it’s flagged as (likely to be) CP, you are not meant to open it. Reviewing content en masse is obviously OK -- but I would not recommend you do that without a PhotoDNA first.
bertil 8y ago
Couple of corrections because I got several things wrong: - credit where credit is due: the technology was developed by Microsoft; - it was developed as a generic image recognition software, hence the neutral name; - it is a hash, not a bloom filter; not sure why I remember it as a bloom; it doesn’t appear to be able to recognise new images; - the technology is widely used, but it was donated by Microsoft to International Centre for Missing & Exploited Children (ICMEC) which isn’t loosely connected but, yes, has close ties with all local law enforcement on those questions. It is available on Azure Marketplace, so widely available. The technology is still being expanded to work on videos and could be used to identify terrorism-related content. Because it’s meant to be resistant to alterations but not infer new content, it is not able to flag new content: that has to be manual. (Someone at the ICMEC has a horrifying job.) I would assume that it is generally implemented by flagging some images in uploaded albums and sending the whole album, or images from the same uploader, for manual verification. That would explain what I’ve heard about some images being potential. To come back to the original story: it seems more likely that Tumblr’s problems are related to new images.
sascha_sl 8y ago
late reply, but I reached out to PhotoDNA at the time after getting the first user report. They told me to fuck off, no individuals wanted. Their site just states you need to "go through a vetting process". Apparently you de-facto need to be a corporation to allow user uploads without getting fucked now.

by bertil 8y ago

ballenf 8y ago

> even opening the file would be illegal Not an expert, but I've worked with large companies that used humans to filter user content. Either they were breaking the law en masse deliberately or there is a safe harbor for doing manual filtering. Again, I don't know the answer, but I'm very skeptical that it's always illegal to manually filter user content. Presumably swift deletion (and cache purging, if possible), reporting and record-keeping would be important. Although that does raise the issue of how you preserve evidence for police to take action if you delete it immediately... geez this is such a landmine.

MrStonedOne 8y ago
There is an affirmative defence in most jurisdictions for reporting cp you come in contact with.
bertil 8y ago
Not an expert either, but I believe that once it’s flagged as (likely to be) CP, you are not meant to open it. Reviewing content en masse is obviously OK -- but I would not recommend you do that without a PhotoDNA first.

bertil 8y ago

Couple of corrections because I got several things wrong: - credit where credit is due: the technology was developed by Microsoft; - it was developed as a generic image recognition software, hence the neutral name; - it is a hash, not a bloom filter; not sure why I remember it as a bloom; it doesn’t appear to be able to recognise new images; - the technology is widely used, but it was donated by Microsoft to International Centre for Missing & Exploited Children (ICMEC) which isn’t loosely connected but, yes, has close ties with all local law enforcement on those questions. It is available on Azure Marketplace, so widely available. The technology is still being expanded to work on videos and could be used to identify terrorism-related content. Because it’s meant to be resistant to alterations but not infer new content, it is not able to flag new content: that has to be manual. (Someone at the ICMEC has a horrifying job.) I would assume that it is generally implemented by flagging some images in uploaded albums and sending the whole album, or images from the same uploader, for manual verification. That would explain what I’ve heard about some images being potential. To come back to the original story: it seems more likely that Tumblr’s problems are related to new images.

sascha_sl 8y ago

late reply, but I reached out to PhotoDNA at the time after getting the first user report. They told me to fuck off, no individuals wanted. Their site just states you need to "go through a vetting process". Apparently you de-facto need to be a corporation to allow user uploads without getting fucked now.