4 ms·

I posted another comment that was misunderstood as well. Folks, no one is proposing to download actual CSAM images to your photo lib. You could be duped thinkin

I posted another comment that was misunderstood as well. Folks, no one is proposing to download actual CSAM images to your photo lib. You could be duped thinking you downloaded an image of a beautiful sunset which was carefully manipulated to match the hash of an actual CSAM image.

onepunchedman 5y ago
This is the really scary part. Of course getting someone to download blobs that corrolate to CSAM would be one thing, but downloading regular photos that have nefarious hashes is a trend /pol/ could start in an afternoon.
- deleted 5y ago
  [deleted]
robertoandred 5y ago
The parent was proposing to “just send known CSAM”. But OK, say someone sends you a sunset that fools the hasher. Then what? Of course one match won’t do anything, so you’d need to download however many matching sunsets. Then what? The Apple reviewer would see they’re sunsets and you’d challenge the flag saying they’re sunsets. And if somehow NCMEC got involved, they’d see they’re just sunsets. And if law enforcement got involved, they’d see they’re just sunsets. These proofs of concept might seem interesting from a ML pov, but all they do is just highlight why Apple put so many layers of checking into this.
- systemvoltage 5y ago
  It would still be mentally draining to be accussed of CP. Can you imaging how terrified one would be if they see a warning message with a blurred sunset? I don't know exactly how the system works but from Apple's press release, it hides the image and gives a warning to the user. This would not go well on social media.
- netr0ute 5y ago
  The point isn't to trick NCMEC, but rather create a DoS attack so no actual triggers can get through the noise.
  - riffraff 5y ago
    But who would want that? We all want privacy but it seems odd to try to DoS this, with high risk for yourself and very little to gain. Might be useful when the system turns into mass political surveillance tho.
  - robertoandred 5y ago
    I thought the point was to SWAT some innocent person? The goal keeps changing.
  - mukesh610 5y ago
    As I've commented elsewhere, DoS can be easily mitigated by implementing another layer with basic object recognition to filter out false positive collisions.
- heavyset_go 5y ago
  > But OK, say someone sends you a sunset that fools the hasher. Then what? Of course one match won’t do anything, so you’d need to download however many matching sunsets. Then what? A real attack would be to take legal porn images and make them collide with illegal images, so when a human goes to review the scaled down derivative images, those images very well look like they could be CSAM. Since there are many of them, they'd get sent to law enforcement. Then law enforcement would raid the victim's home and take all of their electronic devices in order to determine if they can be charged with a crime or not.
  - abraae 5y ago
    This where the "fog of war" kicks in. What with doors being busted down, police departments making press releases, etc. I can easily imagine that the victim could be prosecuted, convicted and sent away because no-one understood the subtlety that their legal porn was not in fact CSAM.
    - simondotau 5y ago
      The fog of war is largely in the realm of post-puberty minors, photos of which are not being included in Apple's corpus of hashes. I find it difficult to believe that anyone could mistake or otherwise "fog of war" a photograph of an adult and a prepubescent minor. And that's assuming someone develops a hash collision which doesn't substantially mangle the photograph like the example offered on Github. Specifically, only images categorised as "A1" are being included in the hash set on iOS. The category definitions are: A = prepubescent minor B = pubescent minor 1 = sex act 2 = "lascivious exhibition" The categories are described in further detail (ugh) in this PDF, page 22: https://www.prosecutingattorneys.org/wp-content/uploads/Presentation-Slides2.pdf https://www.prosecutingattorneys.org/wp-content/uploads/Pres...
      - Syonyk 5y ago
        > Specifically, only images categorised as "A1" are being included in the hash set on iOS. Do we know that for sure? Apple has changed their mind enough times in the last week and a half that I'm convinced they're in full on defensive "wing it and say whatever will get people off our backs!" mode. You can't read the threat modeling PDF and conclude that it was run through the normal Apple document review process. It reads nothing like a standard Apple document - it reads like a bunch of sleep deprived people were told to whip it up and publish it.
- kuratkull 5y ago
  Remember, while you are refuting all this to each party, you are actually in the process of defending yourself against one of the worst criminal accusations possible. Your life will be investigated, your devices will be investigated - the amount of stress and reputational harm this causes is insane.
arsome 5y ago
The even worse part here is that not only could it impact an image of a beautiful sunset, which would fail the human check, it could impact a low quality version of legal porn, which could easily pass the human check and get passed on to law enforcement. A sufficiently advanced catfishing attack could probably take advantage of this to get someone raided and have all their electronics confiscated. Just send someone a zip of photos and let them extract it...
zepto 5y ago
Yes but that would be harmless because the visual derivative wouldn’t match.
simondotau 5y ago
> You could be duped thinking you downloaded an image of a beautiful sunset If it was anything like the image used to demonstrate this technique on Github, it's unlikely that anyone would describe that sunset as "beautiful". They'd be more likely to describe it as "bugger, this JPEG file is corrupted."
- Syonyk 5y ago
  Attacks never get worse over time. It was quite literally less than 24h from "Oh, hey, I can collide this grey blob with a dog!" to "Hey, this thing that looks like cat hashes to the same thing as this dog!" You really think this is going to end at this proof of concept stage?
  - simondotau 5y ago
    Of course it will get better. But it's not going to end at "Hey, this photograph of a sunset is visually unchanged" while now matching CSAM. That's just not plausible. It's not how these classifiers work. Regardless, this whole thing is moot because there are two classifiers, only one of which has been made public. Before any matches can make it to human review, photos in decrypted vouchers have to pass the CSAM match against a second classifier that Apple keeps to itself.
    - copperx 5y ago
      Match the first classifier, and your file gets uploaded unencrypted to Apple. Which is fine if it's probable CSAM. But what if they switch efforts to combat, say, piracy?
      - jdlshore 5y ago
        That’s not how the technology works. The files are never decrypted. Instead, if enough hashes match, a “visual derivative” is revealed. What a “visual derivative” is hasn’t been explained, but most people seem to think it’s a low-res version of the file.
      - 5y ago

onepunchedman 5y ago

This is the really scary part. Of course getting someone to download blobs that corrolate to CSAM would be one thing, but downloading regular photos that have nefarious hashes is a trend /pol/ could start in an afternoon.

deleted 5y ago
[deleted]

robertoandred 5y ago

The parent was proposing to “just send known CSAM”. But OK, say someone sends you a sunset that fools the hasher. Then what? Of course one match won’t do anything, so you’d need to download however many matching sunsets. Then what? The Apple reviewer would see they’re sunsets and you’d challenge the flag saying they’re sunsets. And if somehow NCMEC got involved, they’d see they’re just sunsets. And if law enforcement got involved, they’d see they’re just sunsets. These proofs of concept might seem interesting from a ML pov, but all they do is just highlight why Apple put so many layers of checking into this.

systemvoltage 5y ago
It would still be mentally draining to be accussed of CP. Can you imaging how terrified one would be if they see a warning message with a blurred sunset? I don't know exactly how the system works but from Apple's press release, it hides the image and gives a warning to the user. This would not go well on social media.
netr0ute 5y ago
The point isn't to trick NCMEC, but rather create a DoS attack so no actual triggers can get through the noise.
- riffraff 5y ago
  But who would want that? We all want privacy but it seems odd to try to DoS this, with high risk for yourself and very little to gain. Might be useful when the system turns into mass political surveillance tho.
- robertoandred 5y ago
  I thought the point was to SWAT some innocent person? The goal keeps changing.
- mukesh610 5y ago
  As I've commented elsewhere, DoS can be easily mitigated by implementing another layer with basic object recognition to filter out false positive collisions.
heavyset_go 5y ago
> But OK, say someone sends you a sunset that fools the hasher. Then what? Of course one match won’t do anything, so you’d need to download however many matching sunsets. Then what? A real attack would be to take legal porn images and make them collide with illegal images, so when a human goes to review the scaled down derivative images, those images very well look like they could be CSAM. Since there are many of them, they'd get sent to law enforcement. Then law enforcement would raid the victim's home and take all of their electronic devices in order to determine if they can be charged with a crime or not.
- abraae 5y ago
  This where the "fog of war" kicks in. What with doors being busted down, police departments making press releases, etc. I can easily imagine that the victim could be prosecuted, convicted and sent away because no-one understood the subtlety that their legal porn was not in fact CSAM.
  - simondotau 5y ago
    The fog of war is largely in the realm of post-puberty minors, photos of which are not being included in Apple's corpus of hashes. I find it difficult to believe that anyone could mistake or otherwise "fog of war" a photograph of an adult and a prepubescent minor. And that's assuming someone develops a hash collision which doesn't substantially mangle the photograph like the example offered on Github. Specifically, only images categorised as "A1" are being included in the hash set on iOS. The category definitions are: A = prepubescent minor B = pubescent minor 1 = sex act 2 = "lascivious exhibition" The categories are described in further detail (ugh) in this PDF, page 22: https://www.prosecutingattorneys.org/wp-content/uploads/Presentation-Slides2.pdf https://www.prosecutingattorneys.org/wp-content/uploads/Pres...
    - Syonyk 5y ago
      > Specifically, only images categorised as "A1" are being included in the hash set on iOS. Do we know that for sure? Apple has changed their mind enough times in the last week and a half that I'm convinced they're in full on defensive "wing it and say whatever will get people off our backs!" mode. You can't read the threat modeling PDF and conclude that it was run through the normal Apple document review process. It reads nothing like a standard Apple document - it reads like a bunch of sleep deprived people were told to whip it up and publish it.
kuratkull 5y ago
Remember, while you are refuting all this to each party, you are actually in the process of defending yourself against one of the worst criminal accusations possible. Your life will be investigated, your devices will be investigated - the amount of stress and reputational harm this causes is insane.

arsome 5y ago

The even worse part here is that not only could it impact an image of a beautiful sunset, which would fail the human check, it could impact a low quality version of legal porn, which could easily pass the human check and get passed on to law enforcement. A sufficiently advanced catfishing attack could probably take advantage of this to get someone raided and have all their electronics confiscated. Just send someone a zip of photos and let them extract it...

zepto 5y ago

Yes but that would be harmless because the visual derivative wouldn’t match.

simondotau 5y ago

> You could be duped thinking you downloaded an image of a beautiful sunset If it was anything like the image used to demonstrate this technique on Github, it's unlikely that anyone would describe that sunset as "beautiful". They'd be more likely to describe it as "bugger, this JPEG file is corrupted."

Syonyk 5y ago
Attacks never get worse over time. It was quite literally less than 24h from "Oh, hey, I can collide this grey blob with a dog!" to "Hey, this thing that looks like cat hashes to the same thing as this dog!" You really think this is going to end at this proof of concept stage?
- simondotau 5y ago
  Of course it will get better. But it's not going to end at "Hey, this photograph of a sunset is visually unchanged" while now matching CSAM. That's just not plausible. It's not how these classifiers work. Regardless, this whole thing is moot because there are two classifiers, only one of which has been made public. Before any matches can make it to human review, photos in decrypted vouchers have to pass the CSAM match against a second classifier that Apple keeps to itself.
  - copperx 5y ago
    Match the first classifier, and your file gets uploaded unencrypted to Apple. Which is fine if it's probable CSAM. But what if they switch efforts to combat, say, piracy?
    - jdlshore 5y ago
      That’s not how the technology works. The files are never decrypted. Instead, if enough hashes match, a “visual derivative” is revealed. What a “visual derivative” is hasn’t been explained, but most people seem to think it’s a low-res version of the file.
    - 5y ago