4 ms·

> This site demonstrates no danger at all. Hash collisions like this are expected. Collisions are expected. Being able to create a preimage on demand is not.

> This site demonstrates no danger at all. Hash collisions like this are expected. Collisions are expected. Being able to create a preimage on demand is not. The entire scheme risks running afoul of Goodhart's Law (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law). By calculating the neuralhash of an image in a publicly-known way (so the calculation can run on the user's device), someone distributing illicit material can modify said material to deliberately collide with a known-good image. This provides a reasonable guarantee that the illicit material is not in fact caught by the on-device scanning, and if the illicit material is later added to the "known-bad" database the collision with the known-good image will decrease the signal/noise ratio of the whole monitoring scheme.

colinmhayes 5y ago
Apple says they're using two hashing algorithms. Even if you could figure out collisions with the hidden one finding a collision with both seems unlikely, although I am looking forward to more rigorous review.
- nullc 5y ago
  If the second hash is just another version of neuralhash it's not that unlikely that the same image could match both. It's photodna, people could make images that match both. You really can't say anything about "unlikely" when the other algorithm is a secret. Perhaps it's just "return true". Apple failed to disclose[*] that neuralhash could be adversarially matched until the public exploited it, what else are they failing to disclose? [*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible. I generated my first match with almost no experience with the machine learning tools in about an hour, most of which was spent trying to figure out how to use python neural network glue stuff. It should have been expected by design, and other that getting lucky with some parameters basically the dumbest thing possible works (not for every image pair, but enough).
  - zepto 5y ago
    > You really can't say anything about "unlikely" when the other algorithm is a secret. Perhaps it's just "return true". You’re obviously not being honest. You don’t believe it could actually be just ‘return true’. > [*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible. Of course they knew it was possible. That’s why the system doesn’t rely on it not being possible.
    - nullc 5y ago
      I believe it is likely that apple will improperly fail to secure the files they can decrypt from insider attack, and I am confident that governments can order them (third party doctrine) to hand over material, bypassing those protections. If you're going to question my honesty, can I ask what financial interest you have in apple or this abuse system, if any? > Of course they knew it was possible. I'm glad we agree. But the fact that their hash is subject to adversarial images, causing totally innocent images to be decryptable by apple, is disclosed nowhere in their documentation or security analysis. I think that is extremely concerning, particularly because it undermines a central plank of their claimed protections: They claim they will protect against state actors including non-child-porn material in their databases by intersecting multiple databases from multiple states. This claimed "protection" is completely obliterated by state actors being able to modify child porn to match the hashes of other images and then submitting this material to each other.
      - zepto 5y ago
        > If you're going to question my honesty… The questioning is a response to your comment. Do you seriously think that Apple’s second hash function could be ‘return true’?
        nullc 5y ago
        Functionally yes: If apple receives an administrative subpoena to turn over the unfiltered initial matches, then they'll be divulged without further processing. Literally, presumably not, otoh they've outright lied about this system already but I wouldn't think it likely. And when your privacy is compromised your situation will not be improved because the material that mislead you happened to be "technically true".

colinmhayes 5y ago

Apple says they're using two hashing algorithms. Even if you could figure out collisions with the hidden one finding a collision with both seems unlikely, although I am looking forward to more rigorous review.

nullc 5y ago
If the second hash is just another version of neuralhash it's not that unlikely that the same image could match both. It's photodna, people could make images that match both. You really can't say anything about "unlikely" when the other algorithm is a secret. Perhaps it's just "return true". Apple failed to disclose[*] that neuralhash could be adversarially matched until the public exploited it, what else are they failing to disclose? [*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible. I generated my first match with almost no experience with the machine learning tools in about an hour, most of which was spent trying to figure out how to use python neural network glue stuff. It should have been expected by design, and other that getting lucky with some parameters basically the dumbest thing possible works (not for every image pair, but enough).
- zepto 5y ago
  > You really can't say anything about "unlikely" when the other algorithm is a secret. Perhaps it's just "return true". You’re obviously not being honest. You don’t believe it could actually be just ‘return true’. > [*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible. Of course they knew it was possible. That’s why the system doesn’t rely on it not being possible.
  - nullc 5y ago
    I believe it is likely that apple will improperly fail to secure the files they can decrypt from insider attack, and I am confident that governments can order them (third party doctrine) to hand over material, bypassing those protections. If you're going to question my honesty, can I ask what financial interest you have in apple or this abuse system, if any? > Of course they knew it was possible. I'm glad we agree. But the fact that their hash is subject to adversarial images, causing totally innocent images to be decryptable by apple, is disclosed nowhere in their documentation or security analysis. I think that is extremely concerning, particularly because it undermines a central plank of their claimed protections: They claim they will protect against state actors including non-child-porn material in their databases by intersecting multiple databases from multiple states. This claimed "protection" is completely obliterated by state actors being able to modify child porn to match the hashes of other images and then submitting this material to each other.
    - zepto 5y ago
      > If you're going to question my honesty… The questioning is a response to your comment. Do you seriously think that Apple’s second hash function could be ‘return true’?
      - nullc 5y ago
        Functionally yes: If apple receives an administrative subpoena to turn over the unfiltered initial matches, then they'll be divulged without further processing. Literally, presumably not, otoh they've outright lied about this system already but I wouldn't think it likely. And when your privacy is compromised your situation will not be improved because the material that mislead you happened to be "technically true".