6 ms·

And they can easily generate false positives! The worst of both worlds.

simondotau 5y ago
You are correct, fooling one of the two hashes Apple places between an image and human review is technically easy, with some prerequisites. (The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges). Talking about how trivial it is on a technical level is, in my opinion, an oversimplification. For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM images. These are images of sexually explicit material where pre-pubescent children are actively involved. Following introspection of my own personal moral compass, it's clear to me that being knowingly in proximity to such material would not be "easy" for me.
- masswerk 5y ago
  However, it may be sufficient to just know the hashes of the offending images – and then craft images accordingly.
  - simondotau 5y ago
    If the target hash is sufficient, then yes, that would invalidate my argument. Although that still requires one to come into possession of such a hash string, which might not be "easy" as a purely practical exercise.
    - nullc 5y ago
      All my preimage attack images were created using only the hash of the target, no access to the target image was needed, nor is it obvious to me how it would be helpful at all (at least assuming the goal was for the false positive to look like some unrelated image).
- edgyquant 5y ago
  But if you are such a person, who these systems are trying to fight against, it is trivial to defeat the detectors. These systems aren’t designed to hurt the average person.
  - simondotau 5y ago
    I agree, building tools which could lightly mangle CSAM enough to change its hash and defeat detection is trivial. Generating false negatives from a hash algorithm is trivial.
- tsimionescu 5y ago
  A false positive for a CSAM detector is, by definition, something that is not CSAM, but the detector thinks it is.
  - simondotau 5y ago
    You are correct that the output of a false positive generator would not be CSAM. But one of the two inputs would need to be CSAM or a derivation of it.
    - ClumsyPilot 5y ago
      hash of CSAM is not CSAM do you really think there won't be a large list of CSAM hashes on torrents if you go look for it?
      - simondotau 5y ago
        I doubt you'd find a list of NeuralHash CSAM hashes. And it's even less likely that you'll find a list which includes both NeuralHash as well as Apple's secondary perceptual hash which has not been made public. But maybe you could. Let's say hypothetically, such a torrent does exist. Apple could just rev their hash algorithm in the next version of iOS, immediately rendering those lists useless.
- nullc 5y ago
  > The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims). > For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images. You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted. You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
  - simondotau 5y ago
    > You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model. What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is: 1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default. 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database. 3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that. 4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them. 5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious. 6. Hope that your target does actually import all thirty images into their iCloud photo gallery. 7. Hope that your hash collsions all work correctly in practice. 8. Hope that Apple's human reviewers screw up. 9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS. 10. Hope that the child safety organisation which Apple reports this to also screws up. 11. Hope that the law enforcement body which this child safety organisation reports this to also screws up. 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target. That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of. Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.

You are correct, fooling one of the two hashes Apple places between an image and human review is technically easy, with some prerequisites. (The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges). Talking about how trivial it is on a technical level is, in my opinion, an oversimplification. For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM images. These are images of sexually explicit material where pre-pubescent children are actively involved. Following introspection of my own personal moral compass, it's clear to me that being knowingly in proximity to such material would not be "easy" for me.

masswerk 5y ago
However, it may be sufficient to just know the hashes of the offending images – and then craft images accordingly.
- simondotau 5y ago
  If the target hash is sufficient, then yes, that would invalidate my argument. Although that still requires one to come into possession of such a hash string, which might not be "easy" as a purely practical exercise.
  - nullc 5y ago
    All my preimage attack images were created using only the hash of the target, no access to the target image was needed, nor is it obvious to me how it would be helpful at all (at least assuming the goal was for the false positive to look like some unrelated image).
edgyquant 5y ago
But if you are such a person, who these systems are trying to fight against, it is trivial to defeat the detectors. These systems aren’t designed to hurt the average person.
- simondotau 5y ago
  I agree, building tools which could lightly mangle CSAM enough to change its hash and defeat detection is trivial. Generating false negatives from a hash algorithm is trivial.
tsimionescu 5y ago
A false positive for a CSAM detector is, by definition, something that is not CSAM, but the detector thinks it is.
- simondotau 5y ago
  You are correct that the output of a false positive generator would not be CSAM. But one of the two inputs would need to be CSAM or a derivation of it.
  - ClumsyPilot 5y ago
    hash of CSAM is not CSAM do you really think there won't be a large list of CSAM hashes on torrents if you go look for it?
    - simondotau 5y ago
      I doubt you'd find a list of NeuralHash CSAM hashes. And it's even less likely that you'll find a list which includes both NeuralHash as well as Apple's secondary perceptual hash which has not been made public. But maybe you could. Let's say hypothetically, such a torrent does exist. Apple could just rev their hash algorithm in the next version of iOS, immediately rendering those lists useless.
nullc 5y ago
> The second hash algorithm is not currently public knowledge; fooling an unknown algorithm presents a different set of challenges At a minimum it is known to the state actors generating these databases. Apple itself has directly acknowledged abuse by state actors as a threat vector, so it is entirely inappropriate to rely on obscurity here. (and would be a bad security practice regardless, since presumably other attackers could steal the other algorithims). > For the purposes of fooling Apple's NeuralHash to incorrectly flag a CSAM false positive, "easily" involves being in possession of the absolute worst-of-the-worst CSAM image This is false. It requires knowing the hash of the target image, which is completely lawful to possess. You don't have to be in possession of the illegal images. You cannot assume these hashes are impossible to obtain-- because if they were it would mean the whole scheme was pointless (since it would also mean the unlawful images were impossible to obtain). But even if they were impossible to obtain, an attacker could simply create new "apparent" child abuse material submit it to the databases while at the same time creating whatever false matches against it that they wanted. You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model.
- simondotau 5y ago
  > You appear to be assuming an attack model where the attackers won't do "bad" things. Under that set of assumptions all systems are secure. Reality disagrees with this model. What I'm saying is that a whole bunch of stars need to align before any practical use for this exploit becomes even remotely plausible. This is slightly more likely if your adversary is a state actor with strong motivations and no issues accessing and handling CSAM. But consider the necessary steps and ask how likely it is: 1. Hope that your client uses an iPhone, pays for iCloud storage, and has enabled the cloud photo library feature, not on by default. 2. Acquire the NeuralHash and secondary hash for thirty A1-class images in the CSAM database. 3. Acquire the secondary hash algorithm and build a tool that generates hash collisions for that. 4. Find thirty images which have even the most vanishing small hope of being accidentally identifiable as CSAM by a human reviewer, and that your target would like so much that they'd be inclined to import all thirty into their iCloud Photo library upon receiving them. 5. Mangle these thirty images to each match both hashes. But without mangling them so much that the target's desire to import them into their iCloud photo gallery diminishes, whether for aesthetic reasons or because the target becomes suspicious. 6. Hope that your target does actually import all thirty images into their iCloud photo gallery. 7. Hope that your hash collsions all work correctly in practice. 8. Hope that Apple's human reviewers screw up. 9. Hope that none of the 30 images cause any of Apple's reviewers to become suspicious, flagging the matter for technical review, causing Apple to blacklist these hashes and rev their hash algorithms in the next version of iOS. 10. Hope that the child safety organisation which Apple reports this to also screws up. 11. Hope that the law enforcement body which this child safety organisation reports this to also screws up. 12. Hope that bumbling actions of law enforcement cause desirable negative consequences to your target. That's a dozen steps, many of which are exceedingly non-trivial. There's probably a dozen more steps I'm missing or unaware of. Upshot—a lot of work for very little outcome. Far more effective would be to break into the guy's house, stuff some printouts of CSAM under their mattress and then make a bunch of anonymous tip-offs to local law enforcement. Or just pay a guy to shoot him.