4 ms·

Maybe I'm completely paranoid here, but given that actual sex offenders commonly seek out ways to be near children, what happens if one or more of them end up i

by tjmc 5y ago

Maybe I'm completely paranoid here, but given that actual sex offenders commonly seek out ways to be near children, what happens if one or more of them end up in Apple's image vetting team? They'd be completely anonymous and fully covered, with an endless pipeline of naked kids images being delivered to them. The idea that if you take a picture of your kid in the bath, it just happens to match a CSAM fingerprint and then gets silently transmitted to anonymous reviewers for "review" is terrifying.

shuckles 5y ago
Your terrifying idea mischaracterizes the nature of false positives. Any photo in your library is equally liable to be a false positive as any other; the perceptual hash is not looking for similar images by the metric of what you find similar (content). That’s also the underlying idea behind why people have been able to turn arbitrary images into adversarial false positives.
- tjmc 5y ago
  So that picture of my driver's license I took for an ID check or that sensitive work document I scanned with my phone are just as likely to be sent? Great.
  - shuckles 5y ago
    The chance that any pictures from your library are revealed at all is at most one in one trillion (mod you not storing CSAM or being attacked by someone trying to plant evidence on you). Contrast this to a server side scanning system where every photo in your library will be accessed with unknown false positive characteristics.
  - simondotau 5y ago
    The image would need to be vaguely similar in terms of gross shapes and arrangement. It's exceedingly unlikely that any CSAM would ever be remotely similar to an ID card or a sheet of paper. If there are ever going to be any "natural" matches to any CSAM hashes, it's probably going to be a photograph of people who are coincidentally in a similar pose at a nearly identical angle and strikingly shading.
    - waz0wski 5y ago
      In the myriad of articles about this systems many issues there have been comments from people who have worked with the NCMEC upstream database and note that it's filled with mundane photos, empty rooms, etc - I think it was in one of the hackerfactor article discussions This entire system is ripe for false positives AND adversarial attacks.
      - shuckles 5y ago
        The NCMEC database is large and graded to distinguish types of photos. There’s evidence in the false positive calculations that Apple is only using a subset, presumably the one where photos are graded as depicting active abuse. It’s not reasonable to dispute the 1 in 1e12 false positive claim on mere speculation.
        brokenmachine 5y ago
        >It’s not reasonable to dispute the 1 in 1e12 false positive claim on mere speculation. It's entirely reasonable. Have you seen https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ ? Extraordinary claims require extraordinary evidence.
        simondotau 5y ago
        Collision attacks make for a fun tech demo, but I've yet to hear anyone suggest any plausible scenario where they could be used against Apple's implementation. It would require absurdly elaborate, Oceans Eleven style espionage to achieve any outcome whatsoever. And it would be immediately apparent to anyone involved that a collision attack was involved. It would be far easier (and far more effective) to just acquire child porn, break into your victim's house, stash physical prints under their mattress, and then contact the police. Furthermore, the website includes numerous misleading statements about Apple's system, or makes critical omissions on the description of Apple's system. Whatever side you're on, misleading arguments should be dismissed for what they are.
        5y ago
simondotau 5y ago
This is a disgusting thought, but hear me out. Perhaps this might actually be a good job to give to a paedophile. Their classifications would probably have a superior false positive rate than someone who is disgusted by the images, and it would all but eliminate any concern about an employee suffering psychological trauma.
- ericwooley 5y ago
  > and it would all but eliminate any concern about an employee suffering psychological trauma. I doubt this. If they were all images that this person happened to be into, maybe... But even then, I think it would likely make their addiction to child porn worse, which is it's own psychological problem that is probably worse for society than the trauma suffered by current employees. What happens when they leave that job, and are used to seeing hundreds of cp images a day? Not to mention, that some of the stuff the scanner would be looking at would probably be horrific, and violent. Looking at that kind of thing all day would probably have similar psychiatric affects on pedophiles and non pedophiles. In the worst case, it might cause some pedophiles to start to like the worse images out of boredom from seeing so much cp. Overall, I'd say this is would just be a bad avenue to go down.
  - simondotau 5y ago
    I thought we agreed that playing violent video games doesn't make people more violent. Isn't this the same thing?
    - ericwooley 5y ago
      It's not the same thing. The argument would have to be: violent video games lead to more violent video games, which, anecdotally, I would probably say they do.
      - simondotau 5y ago
        The concern is that child pornography leads to more children in danger. This is akin to violent video games lead to violence. We don't ban child pornography because it leads to more viewing of child pornography. We ban it because we recognise that demand for CSAM breeds supply of new CSAM. Also because it is—by definition—the product of child abuse.

by tjmc 5y ago

shuckles 5y ago

Your terrifying idea mischaracterizes the nature of false positives. Any photo in your library is equally liable to be a false positive as any other; the perceptual hash is not looking for similar images by the metric of what you find similar (content). That’s also the underlying idea behind why people have been able to turn arbitrary images into adversarial false positives.

tjmc 5y ago
So that picture of my driver's license I took for an ID check or that sensitive work document I scanned with my phone are just as likely to be sent? Great.
- shuckles 5y ago
  The chance that any pictures from your library are revealed at all is at most one in one trillion (mod you not storing CSAM or being attacked by someone trying to plant evidence on you). Contrast this to a server side scanning system where every photo in your library will be accessed with unknown false positive characteristics.
- simondotau 5y ago
  The image would need to be vaguely similar in terms of gross shapes and arrangement. It's exceedingly unlikely that any CSAM would ever be remotely similar to an ID card or a sheet of paper. If there are ever going to be any "natural" matches to any CSAM hashes, it's probably going to be a photograph of people who are coincidentally in a similar pose at a nearly identical angle and strikingly shading.
  - waz0wski 5y ago
    In the myriad of articles about this systems many issues there have been comments from people who have worked with the NCMEC upstream database and note that it's filled with mundane photos, empty rooms, etc - I think it was in one of the hackerfactor article discussions This entire system is ripe for false positives AND adversarial attacks.
    - shuckles 5y ago
      The NCMEC database is large and graded to distinguish types of photos. There’s evidence in the false positive calculations that Apple is only using a subset, presumably the one where photos are graded as depicting active abuse. It’s not reasonable to dispute the 1 in 1e12 false positive claim on mere speculation.
      - brokenmachine 5y ago
        >It’s not reasonable to dispute the 1 in 1e12 false positive claim on mere speculation. It's entirely reasonable. Have you seen https://thishashcollisionisnotporn.com/ https://thishashcollisionisnotporn.com/ ? Extraordinary claims require extraordinary evidence.
        simondotau 5y ago
        Collision attacks make for a fun tech demo, but I've yet to hear anyone suggest any plausible scenario where they could be used against Apple's implementation. It would require absurdly elaborate, Oceans Eleven style espionage to achieve any outcome whatsoever. And it would be immediately apparent to anyone involved that a collision attack was involved. It would be far easier (and far more effective) to just acquire child porn, break into your victim's house, stash physical prints under their mattress, and then contact the police. Furthermore, the website includes numerous misleading statements about Apple's system, or makes critical omissions on the description of Apple's system. Whatever side you're on, misleading arguments should be dismissed for what they are.
        5y ago

simondotau 5y ago

This is a disgusting thought, but hear me out. Perhaps this might actually be a good job to give to a paedophile. Their classifications would probably have a superior false positive rate than someone who is disgusted by the images, and it would all but eliminate any concern about an employee suffering psychological trauma.

ericwooley 5y ago
> and it would all but eliminate any concern about an employee suffering psychological trauma. I doubt this. If they were all images that this person happened to be into, maybe... But even then, I think it would likely make their addiction to child porn worse, which is it's own psychological problem that is probably worse for society than the trauma suffered by current employees. What happens when they leave that job, and are used to seeing hundreds of cp images a day? Not to mention, that some of the stuff the scanner would be looking at would probably be horrific, and violent. Looking at that kind of thing all day would probably have similar psychiatric affects on pedophiles and non pedophiles. In the worst case, it might cause some pedophiles to start to like the worse images out of boredom from seeing so much cp. Overall, I'd say this is would just be a bad avenue to go down.
- simondotau 5y ago
  I thought we agreed that playing violent video games doesn't make people more violent. Isn't this the same thing?
  - ericwooley 5y ago
    It's not the same thing. The argument would have to be: violent video games lead to more violent video games, which, anecdotally, I would probably say they do.
    - simondotau 5y ago
      The concern is that child pornography leads to more children in danger. This is akin to violent video games lead to violence. We don't ban child pornography because it leads to more viewing of child pornography. We ban it because we recognise that demand for CSAM breeds supply of new CSAM. Also because it is—by definition—the product of child abuse.