3 ms·

I created a proof of concept showing how OpenAI's CLIP model can function as a "sanity check" similar to how Apple says their server-side model works. In order

I created a proof of concept showing how OpenAI's CLIP model can function as a "sanity check" similar to how Apple says their server-side model works. In order for a collision to get through to the human checkers, the same image would have to fool both networks independently: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/

mrits 5y ago
Cool project. Wouldn't padding all generated images with a few items that match CLIP get around this though?
- yeldarb 5y ago
  Doing that would mean the NeuralHash would change though. And you'd have to not only get CLIP to identify CSAM in the generated image but also negate the parts of the generated image that are causing CLIP to label it as "generated" (while still colliding with the target NeuralHash). Unclear how hard this would actually be in practice (if I were going to attempt it, the first thing I'd try is to evolve a colliding image with something like CLIP+VQGAN) but certainly harder than finding a collision alone.
  - salawat 5y ago
    Take 2 CSAM pictures, known. Combine into one image. Swing and a miss. Not in the CSAM dataset. Take two images. Encode alternating pixels. Decode to get the original image back. Convert to different encodings print to PDF or Postscript. Encode as base64 representations of the image file... Who are we trying to fool here? This is kiddie stuff. This is more about trying to implant scanning capabilities on client devices.
    - yeldarb 5y ago
      Those are examples of triggering false negatives (CSAM that gets missed by the system), this collision attack is about triggering false positives (non-CSAM that triggers human intervention).
    - rootusrootus 5y ago
      > This is more about trying to implant scanning capabilities on client devices. Did we think they didn't already have that ability?
      - salawat 5y ago
        No. The brazen part of this is Apple trying to sell the public on this being okay, effective to the point of mitigating the abuse potential. False negative AND false positive have to counterweight the existence of the vector for abuse. That's what you're measuring against. You can't even guarantee that. This isn't even close to a safe or effective move. The risk management on this is hosed.

mrits 5y ago

Cool project. Wouldn't padding all generated images with a few items that match CLIP get around this though?

yeldarb 5y ago
Doing that would mean the NeuralHash would change though. And you'd have to not only get CLIP to identify CSAM in the generated image but also negate the parts of the generated image that are causing CLIP to label it as "generated" (while still colliding with the target NeuralHash). Unclear how hard this would actually be in practice (if I were going to attempt it, the first thing I'd try is to evolve a colliding image with something like CLIP+VQGAN) but certainly harder than finding a collision alone.
- salawat 5y ago
  Take 2 CSAM pictures, known. Combine into one image. Swing and a miss. Not in the CSAM dataset. Take two images. Encode alternating pixels. Decode to get the original image back. Convert to different encodings print to PDF or Postscript. Encode as base64 representations of the image file... Who are we trying to fool here? This is kiddie stuff. This is more about trying to implant scanning capabilities on client devices.
  - yeldarb 5y ago
    Those are examples of triggering false negatives (CSAM that gets missed by the system), this collision attack is about triggering false positives (non-CSAM that triggers human intervention).
  - rootusrootus 5y ago
    > This is more about trying to implant scanning capabilities on client devices. Did we think they didn't already have that ability?
    - salawat 5y ago
      No. The brazen part of this is Apple trying to sell the public on this being okay, effective to the point of mitigating the abuse potential. False negative AND false positive have to counterweight the existence of the vector for abuse. That's what you're measuring against. You can't even guarantee that. This isn't even close to a safe or effective move. The risk management on this is hosed.