17 ms·

Apple's scheme includes operators manually verifying a low-res version of each image matching CSAM databases before any intervention. Of course, grey noise will

Apple's scheme includes operators manually verifying a low-res version of each image matching CSAM databases before any intervention. Of course, grey noise will never pass for CSAM and will fail that step. The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. The real challenge is generating a real image that could be mistaken for CSAM at low res + is actually benign (or else just send CSAM directly) + matches the hash of real CSAM. This is why SHAttered [1] was such a big deal, but daily random SHA collisions aren't. [1] https://shattered.io/ https://shattered.io/

varispeed 5y ago
> Apple's scheme includes operators manually verifying a low-res version of each image The reviewer, likely on a minimum wage, will report images just in case. Nobody would like to be dragged through the mud because they didn't report something they thought it is innocent.
lifthrasiir 5y ago
But you can essentially perform DoS attack to human checkers, effectively rendering the entire system grind to a halt. The entire system is too reliant on the performance of NeuralHash which can be defaced in many ways. [1] (Added later:) I should note that the DoS attack is only possible with the preimage attack and not the second preimage attack as the issue seemingly suggests, because you need the original CSAM to perform the second preimage attack. But given the second preimage attack is this easy, I don't have any hope for the preimage resistance anyway. (Added much later:) And I realized that Apple did think of this possibility and only stores blinded hashes in the device, so the preimage attack doesn't really work as is. But it seems that the hash output is only 96 bits long according to the repository, so this attack might still be possible albeit with much higher computational cost. [1] To be fair, I don't think that Apple's claim of 1/1,000,000,000,000 false positive rate refers to that of the algorithm. Apple probably tweaked the threshold for manual checking to match that target rate, knowing NeuralHash's false positive rate under the normal circumstances. Of course we know that there is no such thing like the normal circumstances.
- shapefrog 5y ago
  I have seen it suggested that everyone should flood the system with flagged images to overwhelm it in protest to this move by apple. Sounds pretty stupid to me to fill your phone with kiddie porn in protest, but you do you internet people.
  - mannerheim 5y ago
    You don't need to do that, just use images that collide with the hashes.
    - deleted 5y ago
      [deleted]
    - deleted 5y ago
      [deleted]
    - robertoandred 5y ago
      How will you know something collides?
      - falcolas 5y ago
        Because the algorithm and list will be on your phone, and can (has, per TFA) be extracted.
        robertoandred 5y ago
        You cannot extract or reverse the CSAM hashes. They've been encrypted and blinded using server-side-only keys. If TFA said that, it's lying.
        falcolas 5y ago
        One does not need to reverse the CSAM hashes to find a collision with a hash. If the evaluation is being done on the phone, including identifying a hash match, the hashes must also be on the phone.
rdli 5y ago
Are you pinning your hopes that a false positive like this will be appropriately caught because of an army of faceless, low wage workers who stare at CSAM cases all day will immediately flag?
- rootusrootus 5y ago
  Apple has pretty deep pockets. Think how much that judgement is going to be when they find themselves in court for letting someone get raided over gray images. Not that it's going to happen, since it would also require NCMEC to think the images match, but whatever. Attack me! Attack me! I want to retire.
  - hda2 5y ago
    > Apple has pretty deep pockets. For now, sure. What happens when their money runs short? What about the other tech companies that will inevitably be forced to deploy this shit? Will they also have Apple's pretty deep pockets? Blind faith in this system will not magically fix how flawed it is nor the abuse and harm it will allow. This is going to hurt a lot of innocent people.
  - brokenmachine 5y ago
    >Attack me! Attack me! I want to retire. If you post your whatsapp address, I'm sure someone will oblige.
nabakin 5y ago
I don't think that is far away either. I won't be surprised if that is achieved within the day, if not sooner. Also, generating images that look the same as the original and yet produce a different hash.
icelancer 5y ago
> Of course, grey noise will never pass for CSAM and will fail that step. Never? You sure that one or more human operators will never make this mistake, dooming someone's life / causing them immense pain?
- shapefrog 5y ago
  I can guarantee nobody will see the inside of a courtroom, on charges of possession and distribution of child porn for possessing multiple images of grey noise (unless there is some steganography going on).
  - notRobot 5y ago
    Many people never see the inside of a courtroom when false or unproven rape accusations are made against them, but their lives still get ruined because of the negative publicity.
    - shapefrog 5y ago
      Are you suggesting that perhaps less people should report rape accusations, because it might be awkward for the accused to get negative publicity? Thats messed up.
    - bscphil 5y ago
      Those cases are not comparable, because the whole reason they have that impact is that the accusations are usually made publicly (because the whole point is to harm the reputation of one's rapist and warn others, should a conviction prove to be impossible), while CSAM review goes through a neural hash privately on your phone, then privately and anonymously through an Apple reviewer, then is privately reviewed at NCMEC (who - I think - have access to the full size image), and only then is turned over to law enforcement (which should also have access to the full image). It only becomes public knowledge if law enforcement then chooses to charge you - and if all that happens on the basis of an obvious adversarial net image, the result is a publicity shitshow for Apple and you become a civil rights hero after your lawyer (even an underpaid overworked public defender should be able to handle this one) demonstrates this. As others have stated in this thread, I think the real failure case is not someone's life getting ruined by claims of CSAM possession somehow resulting from a bad hash match, but the fact that planted material (or sent via message) can now easily ruin your life because it gets automatically reported; you can't simply delete it and move on any more.
bo1024 5y ago
> The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. Strongly disagree. (1) The primary feature of any decent hash function is that this should not happen. (2) Any preimage attack opens the way for further manipulations like you describe.
- brokensegue 5y ago
  cryptographic hashes are different from image fingerprints
  - bo1024 5y ago
    That's true, one way to put it is that traditionally non-cryptographic hashes are supposed to prevent accidental collisions, while cryptographic ones should prevent even collisions on purpose. But hashing is used in many places that could be vulnerable to an attack, so I think the distinction is blurry. People used MD5 for lots of things but are moving away for this reason, even though they're not in cryptographic settings.
Majromax 5y ago
> The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. It is, actually. Remember that hashes are supposed to be many-bit digests of the original; it should take O(2^256) work to find a message with a chosen 256-bit hash and O(2^128) work to find a "birthday attack" collision. Finding any collision at all with NeuralHash so soon after its release is very surprising, suggesting the algorithm is not very strong. SHAttered is a big deal because it is a fully working attack model, but the writing was on the wall for SHA-1 after the collisions were found in reduced-round variations of the hash. Attacks against an algorithm only get better with time, never worse. Moreover, the break of NeuralHash may be even stronger than the SHAttered attack. The latter modifies two documents to produce a collision, but the NeuralHash collision here may be a preimage attack. It's not clear if the attacker crafted both images to produce the collision or just the second one.
- Ajedi32 5y ago
  NeuralHash is a perceptual hash, not a cryptographically secure hash. Perceptual hashes have trivially findable second preimages by design, as the entire point is for two different images which appear visually similar to return the same result. It's not particularly surprising to me that a perceptual hash might also have collisions that don't look similar to the human eye, though if Apple ever claimed otherwise this counterexample is solid proof that they're wrong.
  - cyanite 5y ago
    The problem is that you’d need the original NeuralHash, which isn’t stored on the device. The device only has a blinded version.
SXX 5y ago
> The real challenge is generating a real image that could be mistaken for CSAM at low res + is actually benign (or else just send CSAM directly) + matches the hash of real CSAM. Why do you have an idea that image have to be benign? Almost everyone watch porn and it's will be so much easier to find collisions by manipulating actual porn images which are not CSAM. Also this way you'll more likely to trigged false-positive from Apple staff since they aren't suppose to see how actual CSAM looks like.

varispeed 5y ago

> Apple's scheme includes operators manually verifying a low-res version of each image The reviewer, likely on a minimum wage, will report images just in case. Nobody would like to be dragged through the mud because they didn't report something they thought it is innocent.

lifthrasiir 5y ago

But you can essentially perform DoS attack to human checkers, effectively rendering the entire system grind to a halt. The entire system is too reliant on the performance of NeuralHash which can be defaced in many ways. [1] (Added later:) I should note that the DoS attack is only possible with the preimage attack and not the second preimage attack as the issue seemingly suggests, because you need the original CSAM to perform the second preimage attack. But given the second preimage attack is this easy, I don't have any hope for the preimage resistance anyway. (Added much later:) And I realized that Apple did think of this possibility and only stores blinded hashes in the device, so the preimage attack doesn't really work as is. But it seems that the hash output is only 96 bits long according to the repository, so this attack might still be possible albeit with much higher computational cost. [1] To be fair, I don't think that Apple's claim of 1/1,000,000,000,000 false positive rate refers to that of the algorithm. Apple probably tweaked the threshold for manual checking to match that target rate, knowing NeuralHash's false positive rate under the normal circumstances. Of course we know that there is no such thing like the normal circumstances.

shapefrog 5y ago
I have seen it suggested that everyone should flood the system with flagged images to overwhelm it in protest to this move by apple. Sounds pretty stupid to me to fill your phone with kiddie porn in protest, but you do you internet people.
- mannerheim 5y ago
  You don't need to do that, just use images that collide with the hashes.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
  - robertoandred 5y ago
    How will you know something collides?
    - falcolas 5y ago
      Because the algorithm and list will be on your phone, and can (has, per TFA) be extracted.
      - robertoandred 5y ago
        You cannot extract or reverse the CSAM hashes. They've been encrypted and blinded using server-side-only keys. If TFA said that, it's lying.
        falcolas 5y ago
        One does not need to reverse the CSAM hashes to find a collision with a hash. If the evaluation is being done on the phone, including identifying a hash match, the hashes must also be on the phone.

rdli 5y ago

Are you pinning your hopes that a false positive like this will be appropriately caught because of an army of faceless, low wage workers who stare at CSAM cases all day will immediately flag?

rootusrootus 5y ago
Apple has pretty deep pockets. Think how much that judgement is going to be when they find themselves in court for letting someone get raided over gray images. Not that it's going to happen, since it would also require NCMEC to think the images match, but whatever. Attack me! Attack me! I want to retire.
- hda2 5y ago
  > Apple has pretty deep pockets. For now, sure. What happens when their money runs short? What about the other tech companies that will inevitably be forced to deploy this shit? Will they also have Apple's pretty deep pockets? Blind faith in this system will not magically fix how flawed it is nor the abuse and harm it will allow. This is going to hurt a lot of innocent people.
- brokenmachine 5y ago
  >Attack me! Attack me! I want to retire. If you post your whatsapp address, I'm sure someone will oblige.

nabakin 5y ago

I don't think that is far away either. I won't be surprised if that is achieved within the day, if not sooner. Also, generating images that look the same as the original and yet produce a different hash.

icelancer 5y ago

> Of course, grey noise will never pass for CSAM and will fail that step. Never? You sure that one or more human operators will never make this mistake, dooming someone's life / causing them immense pain?

shapefrog 5y ago
I can guarantee nobody will see the inside of a courtroom, on charges of possession and distribution of child porn for possessing multiple images of grey noise (unless there is some steganography going on).
- notRobot 5y ago
  Many people never see the inside of a courtroom when false or unproven rape accusations are made against them, but their lives still get ruined because of the negative publicity.
  - shapefrog 5y ago
    Are you suggesting that perhaps less people should report rape accusations, because it might be awkward for the accused to get negative publicity? Thats messed up.
  - bscphil 5y ago
    Those cases are not comparable, because the whole reason they have that impact is that the accusations are usually made publicly (because the whole point is to harm the reputation of one's rapist and warn others, should a conviction prove to be impossible), while CSAM review goes through a neural hash privately on your phone, then privately and anonymously through an Apple reviewer, then is privately reviewed at NCMEC (who - I think - have access to the full size image), and only then is turned over to law enforcement (which should also have access to the full image). It only becomes public knowledge if law enforcement then chooses to charge you - and if all that happens on the basis of an obvious adversarial net image, the result is a publicity shitshow for Apple and you become a civil rights hero after your lawyer (even an underpaid overworked public defender should be able to handle this one) demonstrates this. As others have stated in this thread, I think the real failure case is not someone's life getting ruined by claims of CSAM possession somehow resulting from a bad hash match, but the fact that planted material (or sent via message) can now easily ruin your life because it gets automatically reported; you can't simply delete it and move on any more.

bo1024 5y ago

> The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. Strongly disagree. (1) The primary feature of any decent hash function is that this should not happen. (2) Any preimage attack opens the way for further manipulations like you describe.

brokensegue 5y ago
cryptographic hashes are different from image fingerprints
- bo1024 5y ago
  That's true, one way to put it is that traditionally non-cryptographic hashes are supposed to prevent accidental collisions, while cryptographic ones should prevent even collisions on purpose. But hashing is used in many places that could be vulnerable to an attack, so I think the distinction is blurry. People used MD5 for lots of things but are moving away for this reason, even though they're not in cryptographic settings.

Majromax 5y ago

> The fact that you can randomly manipulate random noise until it matches the hash of an arbitrary image is not surprising. It is, actually. Remember that hashes are supposed to be many-bit digests of the original; it should take O(2^256) work to find a message with a chosen 256-bit hash and O(2^128) work to find a "birthday attack" collision. Finding any collision at all with NeuralHash so soon after its release is very surprising, suggesting the algorithm is not very strong. SHAttered is a big deal because it is a fully working attack model, but the writing was on the wall for SHA-1 after the collisions were found in reduced-round variations of the hash. Attacks against an algorithm only get better with time, never worse. Moreover, the break of NeuralHash may be even stronger than the SHAttered attack. The latter modifies two documents to produce a collision, but the NeuralHash collision here may be a preimage attack. It's not clear if the attacker crafted both images to produce the collision or just the second one.

Ajedi32 5y ago
NeuralHash is a perceptual hash, not a cryptographically secure hash. Perceptual hashes have trivially findable second preimages by design, as the entire point is for two different images which appear visually similar to return the same result. It's not particularly surprising to me that a perceptual hash might also have collisions that don't look similar to the human eye, though if Apple ever claimed otherwise this counterexample is solid proof that they're wrong.
- cyanite 5y ago
  The problem is that you’d need the original NeuralHash, which isn’t stored on the device. The device only has a blinded version.

SXX 5y ago

> The real challenge is generating a real image that could be mistaken for CSAM at low res + is actually benign (or else just send CSAM directly) + matches the hash of real CSAM. Why do you have an idea that image have to be benign? Almost everyone watch porn and it's will be so much easier to find collisions by manipulating actual porn images which are not CSAM. Also this way you'll more likely to trigged false-positive from Apple staff since they aren't suppose to see how actual CSAM looks like.