3 ms·

So, the presumed attack (not against individuals, but to defeat the system) is 1. Identify some innocuous pictures that many many people have (memes, Beyoncé,

by FabHK 5y ago

So, the presumed attack (not against individuals, but to defeat the system) is 1. Identify some innocuous pictures that many many people have (memes, Beyoncé, whatever). 2. Produce CSAM. 3. Mangle it such that it is still CSAM visually, but NeuralHash-collides with the innocuous pictures from step 1. 4. Distribute. 5. Wait until they are (via some other mechanism) a) identified as CSAM, b) added to the NCMEC database, c) added to the Apple on-device database of blinded hashes in some iOS update. 6. Millions of people are suddenly incorrectly flagged for exceeding the threshold by NeuralHash (since they have the innocuous pictures in their library), and the review teams are flooded and can't pick out the small number of actual CSAM holders. That is not without a certain elegance. However, it seems to me that A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much. B) it would be quickly defeated by amending the 2nd tier algorithm (between NeuralHash and human review), though, as you highlight, that might be tricky given that the team working on this presumably only has access to the innocuous false positive collision image, not the (purposefully mangled) CSAM.

saithound 5y ago
> A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much. Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.
SiempreViernes 5y ago
There's also the _very_ important assumption that Apple doesn't check for collisions with popular images before adding new things to the database.
nullc 5y ago
> it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much. You can. Here is an example I created (with links to more): https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise. I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.
- FabHK 5y ago
  > Please explain to me why you posted suggesting otherwise. a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better. I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images). Very impressive work I wasn't aware of.
  - nullc 5y ago
    Great answer. Thank you!

by FabHK 5y ago

saithound 5y ago

> A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much. Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.

SiempreViernes 5y ago

There's also the _very_ important assumption that Apple doesn't check for collisions with popular images before adding new things to the database.

nullc 5y ago

> it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much. You can. Here is an example I created (with links to more): https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise. I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.

FabHK 5y ago
> Please explain to me why you posted suggesting otherwise. a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better. I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images). Very impressive work I wasn't aware of.
- nullc 5y ago
  Great answer. Thank you!