5 ms·

I think I am in dire need of some education here and so I have questions: * Is this a problem with Apple's CSAM discriminator engine or with the fact that it's

I think I am in dire need of some education here and so I have questions: * Is this a problem with Apple's CSAM discriminator engine or with the fact that it's happening on-device? * Would this attack not be possible if scanning was instead happening in the cloud, using the same model? * Are other services (Google Photos, Facebook, etc.) that store photos in the cloud not doing something similar to uploaded photos, with models that may be similarly vulnerable to this attack? I know that an argument against on-device scanning is that people don't like to feel like the device that they own is acting against them - like it's snitching on them. I can understand and actually sympathise with that argument, it feels wrong. But we have known for a long time that computer vision can be fooled with adversarial images. What is special about this particular example? Is it only because it's specifically tricking the Apple CSAM system, which is currently a hotly-debated topic, or is there something particularly bad here, something that is not true with other CSAM "detectors"? I genuinely don't know enough about this subject to comment with anything other than questions.

bo1024 5y ago
Not complete answers but background: apple’s system works by having your device create a hash of each image you have. The hash (a short hexadecimal string) is compared to a list of known CP image hashes, and if it matches, then your image is uploaded to Apple for further investigation. A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. That appears to be what we are witnessing.
- magpi3 5y ago
  But how would the attacker get the generated image on a person's phone?
  - Koffiepoeder 5y ago
    E.g. send them a whatsapp message that looks innocent
    - endisneigh 5y ago
      They can see the image - why would they import a random image into their library from someone they don’t know?
      - markus92 5y ago
        Auto import could be turned on?
      - kemayo 5y ago
        WhatsApp has a really weird default behavior: it imports all images you're sent into your photo library. This is a smart thing to disable, even outside this recent discussion of CSAM. https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp-media-to-your-phones-gallery/?lang=en https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp...
        endisneigh 5y ago
        In that scenario this attack is unnecessary. Someone could just send you legitimate child pornography and then immediately tell the authorities that you possess said things.
        kemayo 5y ago
        Yup! It's a legitimately crazy default. Edit: Though, to be fair, the specific hash-collision scenario would be that someone could send you something that doesn't look like CSAM and so you wouldn't reflexively delete it.
        endisneigh 5y ago
        If it doesn’t look like it wouldn’t a human reviewer disregard it once it gets to that point? Personally I don’t really see the issue.
        5y ago
  - bo1024 5y ago
    I didn't mean to suggest a targeted attack. If the goal is to just overwhelm Apple's system, the attacker doesn't need to target a particular phone, they just need to distribute lots of colliding images to some phones. Even their own phones, since the colliding images wouldn't be illegal.
  - bumblebritches5 5y ago
    Seriously? Look into Pegasus and Candiru
- Edd314159 5y ago
  > A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. This is my understanding too. But is this not also true for other (cloud-based) CSAM scanning systems? Why is Apple's special in this regard?
  - floatingatoll 5y ago
    There’s a lot of overlap between Apple product buyers and “fight the thoughtcrime slippery slope” hackers. The CSAM abusers are presumably (if they’re not stupid) also fanning the flames on that slippery slope perception, because it’s to their benefit if the hackers defeat Apple.
  - bo1024 5y ago
    I don't know. I don't know what other systems use, I just know what I've read recently about Apple's.
    - cyanite 5y ago
      This pretty much sums this entire drama up, I think.
  - nonbirithm 5y ago
    They aren't. Apple could have saved themselves so much backlash and not have caused the outrage to be focused exclusively on them if they hadn't tried to be novel with their method of hashing, and had just announced that they were about to do exactly what all the other tech companies had already been doing for years - server side scanning. Apple would still be accused of walking back on its claims of protecting users' privacy, but for a different reason - by trying to conform. Instead of wasting all the debate on how Apple and only Apple is violating everyone's privacy with its on-device scanning mechanism, which was without precedent, this could have been an educational experience for many people about how little privacy is valued in the cloud in general, no matter who you choose to give your data to, because there is precedent for such privacy violations that take place on the server. Apple could have been just one of the companies in a long line of others whose data management policies would have received significant renewed attention as a result of this. Instead, everyone is focused on criticizing Apple. There is a significant problem with people's perception of "privacy" in tech if merely moving the scan on-device causes this much backlash while those same people stayed silent during the times that Google and Facebook and the rest adopted the very same technique on the server in the past decade. Maybe if Apple had done the same, they would have been able to get away with it.
- rollinggoron 5y ago
  But how does this exact attack and scenario not also apply to Google, Facebook, Microsoft etc... who are also doing the same thing on their clouds servers?
  - bo1024 5y ago
    I don't know what those companies do, hopefully someone who does know will chime in and answer.
    - cyanite 5y ago
      They essentially do the same, just in the cloud meaning they access all images directly.
- fortenforge 5y ago
  No, I don't think that's what we're witnessing. No one has yet demonstrated that they can take a hash alone and produce an image that matches the hash. It's true that that would be bad, but right now you still need the original colliding image.
- cyanite 5y ago
  That’s an oversimplified and misleading description of how the system works, but ok. I recommend reading the technical description, or even the paper linked from that: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
gitgud 5y ago
Basically you are right. There's nothing that special about hash collisions with image recognition. I think this is blowing up because it's cathartic to see a technology you disagree with get undermined and basically broken by the community...

bo1024 5y ago

Not complete answers but background: apple’s system works by having your device create a hash of each image you have. The hash (a short hexadecimal string) is compared to a list of known CP image hashes, and if it matches, then your image is uploaded to Apple for further investigation. A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. That appears to be what we are witnessing.

magpi3 5y ago
But how would the attacker get the generated image on a person's phone?
- Koffiepoeder 5y ago
  E.g. send them a whatsapp message that looks innocent
  - endisneigh 5y ago
    They can see the image - why would they import a random image into their library from someone they don’t know?
    - markus92 5y ago
      Auto import could be turned on?
    - kemayo 5y ago
      WhatsApp has a really weird default behavior: it imports all images you're sent into your photo library. This is a smart thing to disable, even outside this recent discussion of CSAM. https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp-media-to-your-phones-gallery/?lang=en https://faq.whatsapp.com/android/how-to-stop-saving-whatsapp...
      - endisneigh 5y ago
        In that scenario this attack is unnecessary. Someone could just send you legitimate child pornography and then immediately tell the authorities that you possess said things.
        kemayo 5y ago
        Yup! It's a legitimately crazy default. Edit: Though, to be fair, the specific hash-collision scenario would be that someone could send you something that doesn't look like CSAM and so you wouldn't reflexively delete it.
        endisneigh 5y ago
        If it doesn’t look like it wouldn’t a human reviewer disregard it once it gets to that point? Personally I don’t really see the issue.
        5y ago
- bo1024 5y ago
  I didn't mean to suggest a targeted attack. If the goal is to just overwhelm Apple's system, the attacker doesn't need to target a particular phone, they just need to distribute lots of colliding images to some phones. Even their own phones, since the colliding images wouldn't be illegal.
- bumblebritches5 5y ago
  Seriously? Look into Pegasus and Candiru
Edd314159 5y ago
> A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. This is my understanding too. But is this not also true for other (cloud-based) CSAM scanning systems? Why is Apple's special in this regard?
- floatingatoll 5y ago
  There’s a lot of overlap between Apple product buyers and “fight the thoughtcrime slippery slope” hackers. The CSAM abusers are presumably (if they’re not stupid) also fanning the flames on that slippery slope perception, because it’s to their benefit if the hackers defeat Apple.
- bo1024 5y ago
  I don't know. I don't know what other systems use, I just know what I've read recently about Apple's.
  - cyanite 5y ago
    This pretty much sums this entire drama up, I think.
- nonbirithm 5y ago
  They aren't. Apple could have saved themselves so much backlash and not have caused the outrage to be focused exclusively on them if they hadn't tried to be novel with their method of hashing, and had just announced that they were about to do exactly what all the other tech companies had already been doing for years - server side scanning. Apple would still be accused of walking back on its claims of protecting users' privacy, but for a different reason - by trying to conform. Instead of wasting all the debate on how Apple and only Apple is violating everyone's privacy with its on-device scanning mechanism, which was without precedent, this could have been an educational experience for many people about how little privacy is valued in the cloud in general, no matter who you choose to give your data to, because there is precedent for such privacy violations that take place on the server. Apple could have been just one of the companies in a long line of others whose data management policies would have received significant renewed attention as a result of this. Instead, everyone is focused on criticizing Apple. There is a significant problem with people's perception of "privacy" in tech if merely moving the scan on-device causes this much backlash while those same people stayed silent during the times that Google and Facebook and the rest adopted the very same technique on the server in the past decade. Maybe if Apple had done the same, they would have been able to get away with it.
rollinggoron 5y ago
But how does this exact attack and scenario not also apply to Google, Facebook, Microsoft etc... who are also doing the same thing on their clouds servers?
- bo1024 5y ago
  I don't know what those companies do, hopefully someone who does know will chime in and answer.
  - cyanite 5y ago
    They essentially do the same, just in the cloud meaning they access all images directly.
fortenforge 5y ago
No, I don't think that's what we're witnessing. No one has yet demonstrated that they can take a hash alone and produce an image that matches the hash. It's true that that would be bad, but right now you still need the original colliding image.
cyanite 5y ago
That’s an oversimplified and misleading description of how the system works, but ok. I recommend reading the technical description, or even the paper linked from that: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

gitgud 5y ago

Basically you are right. There's nothing that special about hash collisions with image recognition. I think this is blowing up because it's cathartic to see a technology you disagree with get undermined and basically broken by the community...