5 ms·

You're adding quite a lot of technobabble gloss to an "attack vector" that boils down to "people can send you images that are visually indistinguishable from kn

You're adding quite a lot of technobabble gloss to an "attack vector" that boils down to "people can send you images that are visually indistinguishable from known CSAM". Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds. While it would be bad to be subjected to such an attack, and there is a small chance it would lead to some kind of interaction with law enforcement, the outcomes you present are just scaremongering and not reasonable.

saithound 5y ago
I suggest you reread the comment, because "people can send you images that are visually indistinguishable from known CSAM" is not what is being said at all. Where did you even get that from? The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread political memes that trigger automated CSAM filters on people's phones just to "pwn the libz". > Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds. No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. E.g. your own photos are not in the NCMEC databases, and you'd have to reveal your own illegal activities to get them in there. You (or malicious political organizations) especially cannot attack and expose "wrongthinking" groups of people by sending them photos of 17.9 year olds.
- ChrisKnott 5y ago
  Can you explain how these theoretical political memes hash-match to an image in the NCMEC database, and then also pass the visual check? > "No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people." Did I say "taking"? I am talking about sending (theoretical) actual images from the NCMEC database. This is functionally identical to the "attack" you describe.
  - saithound 5y ago
    Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order. You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm. Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm. You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold. Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive. [1] https://github.com/anishathalye/neural-hash-collider https://github.com/anishathalye/neural-hash-collider [2] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/ https://embracethered.com/blog/posts/2020/husky-ai-image-res... [3] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar... [4] https://graphicdesign.stackexchange.com/questions/106260/image-that-looks-completely-different-when-resized https://graphicdesign.stackexchange.com/questions/106260/ima...
    - ChrisKnott 5y ago
      Ok yeah, I do agree this scaling attack potentially makes this feasible, if it essentially allows you to present a completely different image to the reviewer as to the user. Has anyone done this yet? i.e. an image that NeuralHashes to a target hash, and also scale-attacks to a target image, but looks completely different. (Perhaps I misunderstood your original post, but this seems to be a completely different scenario to the one you originally described with reference to the three thumbnails)
- Spooky23 5y ago
  > No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. An attacker can embed a matching image inside of a PowerPoint zip file, and email it to any corporate employee using O365. Or, an angry parent can call the police and let them know that a 16 year old possesses nose pictures of their 15 year old girlfriend. The over top response to this controversy is really disappointing.
  - saithound 5y ago
    Sure, your proposed attack, that requires the victim to have a 15 year old girlfriend, to break an (admittedly silly) law by having nude photos on their phone, for you to call the cops, and for them to take such a call seriously is clearly comparable to a vector that can be used to target innocents, groups of individuals, etc. who did not break the law in any way, and that do not require the attacker to handle prohibitex material at all, and requires Apple to keep a ton of information completely obscure to even provide a weak semblance of security (it was shown to be completely broken except possibly for one unknown hash, in two weeks). Clearly comparable. Sure. Clearly. For one last time, the NeuralHash collisions make this tool perfectly unusable for catching pedos: all of the next generation of CSAM content will collide with hashes of popular, innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. It's completely useless for its nominal function. This would be a massive, hilarious own goal from Apple even if the public reaction was over the top (although it isn't). They just reduced the privacy and security of nearly all their customers, further exposed themselves to the whims of governments, and for no gain whatsoever.

saithound 5y ago

I suggest you reread the comment, because "people can send you images that are visually indistinguishable from known CSAM" is not what is being said at all. Where did you even get that from? The point is precisely that people can become victims of various new attacks, without ever touching photos that are actual "known CSAM". For Christ's sake, half the comments here are about how adversaries can create and spread political memes that trigger automated CSAM filters on people's phones just to "pwn the libz". > Guess what, they can already do this but worse by just sending you actual illegal images of 17.9 year olds. No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. E.g. your own photos are not in the NCMEC databases, and you'd have to reveal your own illegal activities to get them in there. You (or malicious political organizations) especially cannot attack and expose "wrongthinking" groups of people by sending them photos of 17.9 year olds.

ChrisKnott 5y ago
Can you explain how these theoretical political memes hash-match to an image in the NCMEC database, and then also pass the visual check? > "No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people." Did I say "taking"? I am talking about sending (theoretical) actual images from the NCMEC database. This is functionally identical to the "attack" you describe.
- saithound 5y ago
  Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order. You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm. Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm. You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold. Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive. [1] https://github.com/anishathalye/neural-hash-collider https://github.com/anishathalye/neural-hash-collider [2] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/ https://embracethered.com/blog/posts/2020/husky-ai-image-res... [3] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar... [4] https://graphicdesign.stackexchange.com/questions/106260/image-that-looks-completely-different-when-resized https://graphicdesign.stackexchange.com/questions/106260/ima...
  - ChrisKnott 5y ago
    Ok yeah, I do agree this scaling attack potentially makes this feasible, if it essentially allows you to present a completely different image to the reviewer as to the user. Has anyone done this yet? i.e. an image that NeuralHashes to a target hash, and also scale-attacks to a target image, but looks completely different. (Perhaps I misunderstood your original post, but this seems to be a completely different scenario to the one you originally described with reference to the three thumbnails)
Spooky23 5y ago
> No, this misses the point completely. You cannot easily trigger any automated systems merely by taking photos of 17.9 year olds and sending them to people. An attacker can embed a matching image inside of a PowerPoint zip file, and email it to any corporate employee using O365. Or, an angry parent can call the police and let them know that a 16 year old possesses nose pictures of their 15 year old girlfriend. The over top response to this controversy is really disappointing.
- saithound 5y ago
  Sure, your proposed attack, that requires the victim to have a 15 year old girlfriend, to break an (admittedly silly) law by having nude photos on their phone, for you to call the cops, and for them to take such a call seriously is clearly comparable to a vector that can be used to target innocents, groups of individuals, etc. who did not break the law in any way, and that do not require the attacker to handle prohibitex material at all, and requires Apple to keep a ton of information completely obscure to even provide a weak semblance of security (it was shown to be completely broken except possibly for one unknown hash, in two weeks). Clearly comparable. Sure. Clearly. For one last time, the NeuralHash collisions make this tool perfectly unusable for catching pedos: all of the next generation of CSAM content will collide with hashes of popular, innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. It's completely useless for its nominal function. This would be a massive, hilarious own goal from Apple even if the public reaction was over the top (although it isn't). They just reduced the privacy and security of nearly all their customers, further exposed themselves to the whims of governments, and for no gain whatsoever.