4 ms·

I wouldn't doubt that Google participates in the National Center for Missing and Exploited Children database of child porn hashes in order to identify and shut

I wouldn't doubt that Google participates in the National Center for Missing and Exploited Children database of child porn hashes in order to identify and shut down accounts with illegal images in their Google Drive or Gmail. If this is the case, I wonder if this opens up an attack vector against a targeted account. If the system is set up a certain way, an email with a illegal image attachment could cause a google account to be closed and purged, with notification to law enforcement. http://www.missingkids.com/Exploitation/Industry http://www.missingkids.com/Exploitation/Industry states "Through the Hash Value Sharing Initiative, U.S. based Electronic Service Providers can partner with NCMEC to receive a list of MD5 hash values which represent the "worst of the worst" images of apparent child pornography." as well as "Major U.S. companies have implemented PhotoDNA, and the technology is helping them identify child pornography images on their servers that may have otherwise gone undetected." Interestingly enough, if a provider is using just the MD5 database and not PhotoDNA, and someone could get one of the MD5 hashes of an illegal image, a bad actor could use a hash collision attach to make an innocuous image have the same MD5 hash. Also, here is an article (from 2014 so it might be out of date) about which technologies google uses to identify child abuse images: http://www.pcworld.com/article/2461400/how-google-handles-child-pornography-in-gmail-search.html http://www.pcworld.com/article/2461400/how-google-handles-ch... From that article... "Google itself compiles a database of images of possible abuse that have been brought to the company’s attention, sources close to the company said. Those images are then reviewed by a human employee. If confirmed to show abuse, the company assigns each image a unique digital fingerprint, which is entered into its database. " That has to be the worst job ever.

userbinator 10y ago
You'd think they would move to a better hash function by now... MD5 has been broken for many years. There's a little program from a few years ago that let you create colliding blocks in seconds on a PC of the time.
- nkozyra 10y ago
  MD5 is long-broken as a security mechanism, but as a broader identifier of hashes, a soft match (or matches with potential false positives) might actually be preferred. In this case it might not inherently be a bad idea to use (as opposed to for password hashing).
  - colejohnson66 10y ago
    But why use MD5 anyways? Google has quite an impressive image searching algorithm. Why not just use that? Using MD5 won't do anything because, almost certainly, a false positive is /not/ the same image. You'd need some kind of rolling hash that gets longer the longer the file is. Then do a soft check on that.
    - deleted 10y ago
      [deleted]
    - j-pb 10y ago
      Because that would mean they have to train their networks on _huge_ amounts of actual child porn.
    - deleted 10y ago
      [deleted]
  - emn13 10y ago
    Actually, precisely for applications such as password hashing MD5 has not been broken. MD5 is broken only in collusion with the author of the original stream to be hashed. For passwords, that would mean an attacker who can pick the password and already has access to the system might be able to pick a password such that other passwords also grant access... who cares? Of course, irrespective of hash function you need to worry about weak passwords, so you'd need to use salting and an expensive stretching function, or only permit auto-generated passwords. But that's really not a weakness in the hashfunction; it's to mitigate weaknesses in the concept of a user-chosen password.
- emn13 10y ago
  MD5 still has not been broken. Certain aspect of MD5 have been broken, but as a one-way hash function it remains unbroken. In particular, if I (a non-hostile party) pick a file to hash and publish that hash with or without the original file, a hostile attacker still cannot find the original file (nor any other stream) to generate that same hash. What has been broken are various scenarios where the attack controls the source file to be hashed. That means that as a digital signature it's no good anymore. I think it's important to emphasize this because in all the news about weaknesses and attacks, people sometimes get the idea that the sky is falling, and that it's hopeless to really stay secure. That kind of thinking quickly leads in security through obscurity. In fact, the sky is most certainly not falling. Even though many attacks are successful in improving on brute force, even ancient deprecated stuff like MD5 has not been fully broken; indeed it's most critical feature remains essentially untouched (according to wikipedia, preimage attacks have managed to reduce the complexity from 2^128 to 2^123.5 or in other words: nobody cares). And even as a signature, if you had picked MD5 and used MD5 before the attack was developed, you can be reasonably confident that that signature remains valid today. So yes: don't develop new software using MD5, and retire software relying on MD5 for signatures. And for other applications, if you can easily migrate away from MD5 (and really, sha1 too) - do so! It's easier to reason about security if you don't need to sweat details like this. But if you have an existing application that can't easily be migrated, then don't panic either, just be aware that you need to more carefully consider what it means to have a matching hash (in short it's either the same file, or a different file the original author wanted to appear similar).
- ErrantX 10y ago
  > You'd think they would move to a better hash function by now MD5 is (relatively) fast, and a (relatively) small datapoint to store. You're definitely wanting to store a lot of data points too; it's so so much easier to scan and de-scope OK images..... The MD5 catalogue's intent is to allow quick identification of suspect material to drop to manual review (little bit of inference there from what I have read about Google, but I'd be shocked if that wasn't true - false positives in this context are not as bad as true positives [of course] but accusing an innocent person of having child porn is damaging). From the GP: >That has to be the worst job ever. It is. I did it for a while working for a computer forensics firm. Not even slightly pleasant. We used to go through millions of images - to be honest, 99.999999999% of it was "normal" (if often slightly wierd) porn. You do become detached from it after a while. To be honest; the chats/emails grooming young people were the worst.

userbinator 10y ago

You'd think they would move to a better hash function by now... MD5 has been broken for many years. There's a little program from a few years ago that let you create colliding blocks in seconds on a PC of the time.

nkozyra 10y ago
MD5 is long-broken as a security mechanism, but as a broader identifier of hashes, a soft match (or matches with potential false positives) might actually be preferred. In this case it might not inherently be a bad idea to use (as opposed to for password hashing).
- colejohnson66 10y ago
  But why use MD5 anyways? Google has quite an impressive image searching algorithm. Why not just use that? Using MD5 won't do anything because, almost certainly, a false positive is /not/ the same image. You'd need some kind of rolling hash that gets longer the longer the file is. Then do a soft check on that.
  - deleted 10y ago
    [deleted]
  - j-pb 10y ago
    Because that would mean they have to train their networks on _huge_ amounts of actual child porn.
  - deleted 10y ago
    [deleted]
- emn13 10y ago
  Actually, precisely for applications such as password hashing MD5 has not been broken. MD5 is broken only in collusion with the author of the original stream to be hashed. For passwords, that would mean an attacker who can pick the password and already has access to the system might be able to pick a password such that other passwords also grant access... who cares? Of course, irrespective of hash function you need to worry about weak passwords, so you'd need to use salting and an expensive stretching function, or only permit auto-generated passwords. But that's really not a weakness in the hashfunction; it's to mitigate weaknesses in the concept of a user-chosen password.
emn13 10y ago
MD5 still has not been broken. Certain aspect of MD5 have been broken, but as a one-way hash function it remains unbroken. In particular, if I (a non-hostile party) pick a file to hash and publish that hash with or without the original file, a hostile attacker still cannot find the original file (nor any other stream) to generate that same hash. What has been broken are various scenarios where the attack controls the source file to be hashed. That means that as a digital signature it's no good anymore. I think it's important to emphasize this because in all the news about weaknesses and attacks, people sometimes get the idea that the sky is falling, and that it's hopeless to really stay secure. That kind of thinking quickly leads in security through obscurity. In fact, the sky is most certainly not falling. Even though many attacks are successful in improving on brute force, even ancient deprecated stuff like MD5 has not been fully broken; indeed it's most critical feature remains essentially untouched (according to wikipedia, preimage attacks have managed to reduce the complexity from 2^128 to 2^123.5 or in other words: nobody cares). And even as a signature, if you had picked MD5 and used MD5 before the attack was developed, you can be reasonably confident that that signature remains valid today. So yes: don't develop new software using MD5, and retire software relying on MD5 for signatures. And for other applications, if you can easily migrate away from MD5 (and really, sha1 too) - do so! It's easier to reason about security if you don't need to sweat details like this. But if you have an existing application that can't easily be migrated, then don't panic either, just be aware that you need to more carefully consider what it means to have a matching hash (in short it's either the same file, or a different file the original author wanted to appear similar).
ErrantX 10y ago
> You'd think they would move to a better hash function by now MD5 is (relatively) fast, and a (relatively) small datapoint to store. You're definitely wanting to store a lot of data points too; it's so so much easier to scan and de-scope OK images..... The MD5 catalogue's intent is to allow quick identification of suspect material to drop to manual review (little bit of inference there from what I have read about Google, but I'd be shocked if that wasn't true - false positives in this context are not as bad as true positives [of course] but accusing an innocent person of having child porn is damaging). From the GP: >That has to be the worst job ever. It is. I did it for a while working for a computer forensics firm. Not even slightly pleasant. We used to go through millions of images - to be honest, 99.999999999% of it was "normal" (if often slightly wierd) porn. You do become detached from it after a while. To be honest; the chats/emails grooming young people were the worst.