5 ms·

Doing so would create a positive that still doesn't pass Apple's human visual check against the (blurred) CSAM content associated with that checksum, and if it

Doing so would create a positive that still doesn't pass Apple's human visual check against the (blurred) CSAM content associated with that checksum, and if it somehow did, it would still then also have to occur at qty.30 or more, and they'd have to pass a human visual check against the (unblurred) CSAM content by one of the agencies in possession of it. It's not possible to spoof that final test unless you possess real CSAM content, at which point you don't need to spoof that final test and you'll end up arrested for possession.

deleted 5y ago
[deleted]
laverya 5y ago
And if you make the second image be actual, 18+ porn? Will Apple be able to tell the difference between that and CSAM after the blur is applied? Bonus points if you match poses, coloration, background, etc.
- floatingatoll 5y ago
  The only way to match poses, coloration, background, etc is to possess illegal CSAM content. If you do so successfully, you will result in your crafted image passing the blur check and reaching the agency that possesses the original image for final verification, where it will immediately fail because it is obviously a replica. You will then trigger that agency leading law enforcement to find the creator of the image, so that they can arrest whoever possesses CSAM content in order to model replicas after it. I think that's a perfectly acceptable outcome, since anyone with the hubris to both possess CSAM content and create replicas of it especially deserves to be arrested. Do you see a more problematic outcome here?
  - laverya 5y ago
    This assumes that it's impossible to reverse the perceptual hashes used here in such a way that you could determine poses and coloration, for one. And in retrospect, you don't need to match that - you just need it to appear obviously pornographic after the blur is applied in order to get past Apple's reviewers. After that, the lucky individual's life is in the hands of the police/prosecutors. (I have to imagine that both real and faked cases will look pretty much like "Your honor/members of the jury, this person's device contained numerous photos matching known CSAM. No, we won't be showing you the pictures. No, the defence can't see them either." Can you imagine a "tough on crime" prosecutor taking the faked case to trial too? Would the police and prosecutors even know it was faked?)
    - floatingatoll 5y ago
      Apple's reviewers are comparing blurred originals to blurred matches. It needs to look like the blurred original associated with the checksum that matched. It is irrelevant whether the blurred match looks pornographic or not.
      - ultrarunner 5y ago
        This has already been demonstrated. https://twitter.com/SarahJamieLewis/status/1428083744280256512 https://twitter.com/SarahJamieLewis/status/14280837442802565...
        floatingatoll 5y ago
        At which point the image will be handed to the relevant CSAM group unblurred, who will do a visual comparison and find out immediately that it’s not a match, and then reject it without invoking law enforcement.
  - ultrarunner 5y ago
    Cops no-knock raid the wrong houses all the time; what makes you think that the procedure you outlined above will be followed without error? Especially against suspected child abusers, from an anti-child-abuse agency? What if the visual check gets accidentally signed off on, or even gets fraudulently marked as positive by a burned out/lazy/competent-but-distracted employee? There are just so many failure modes for this dragnet that all end with innocent people suffering a very difficult process, even if they don't eventually land in prison. I don't think it's unreasonable to not want to be volunteered to participate.
  - SXX 5y ago
    Why would attacker need to create anything from scratch? If you want to build a dataset of images with people with specific skin and hair color, body types, etc in very specific poses it's can be quite hard and expensive to do so. Because even photo stocks have limited number of such photos. Unless... Unless you're looking to build a porn dataset and you're don't care about copyright. Porn is industry where exabytes of material are produced and published on internet almost every week. Who will agency come to? To some OnlyFans creators?
treesprite82 5y ago
There are concerns without the images needing to make it all the way through their CSAM process. Apple is a US company with obligations to report certain crimes they become aware of, and NCMEC is a US government-tied organisation that dismissed privacy concerns as "the screeching voices of the minority". Consider if the honeypot images (manipulated to match CSAM hashes) are terrorist recruitment material for example.
int_19h 5y ago
At the point where it is submitted for a human visual check, your privacy has already been violated.
- floatingatoll 5y ago
  My privacy is violated every time I leave my home. Anyone can take a photo of me and look up the FBI Most Wanted and see if I’m there. If they think someone is me and they’re wrong, they can still summon law enforcement, and I’ll still be mistreated for their poor judgement. Is this just as unacceptable as the CSAM scanning? Should all public photography be banned, in order to reduce the risk of false positive identifications of innocent people as criminals to zero? Or is that an acceptable degree of privacy impingement for the good of society? Is Apple’s implementation an acceptable trade of impingement and risk, for good for society? We do live in a society, and so zero impingement upon privacies is never going to be acceptable (sorry, free/tech/libs). But instead of discussing whether Apple’s approach violates privacy minimally or not in order to get the job done, these discussions here just keep circling the drain of “putting my privacy at risk by any degree is never acceptable”, when that drain is cemented shut by the existence of society and will never lead to a valid outcome.
  - int_19h 5y ago
    By leaving your home and going out in public, you are implicitly consenting to being observed. So, no, that's not a violation of your privacy. Apple's implementation is more like a stranger sneaking into your house if they suspect you're up to something bad. That would be equally unacceptable - we have warrants for a reason.

deleted 5y ago

[deleted]

laverya 5y ago

And if you make the second image be actual, 18+ porn? Will Apple be able to tell the difference between that and CSAM after the blur is applied? Bonus points if you match poses, coloration, background, etc.

floatingatoll 5y ago
The only way to match poses, coloration, background, etc is to possess illegal CSAM content. If you do so successfully, you will result in your crafted image passing the blur check and reaching the agency that possesses the original image for final verification, where it will immediately fail because it is obviously a replica. You will then trigger that agency leading law enforcement to find the creator of the image, so that they can arrest whoever possesses CSAM content in order to model replicas after it. I think that's a perfectly acceptable outcome, since anyone with the hubris to both possess CSAM content and create replicas of it especially deserves to be arrested. Do you see a more problematic outcome here?
- laverya 5y ago
  This assumes that it's impossible to reverse the perceptual hashes used here in such a way that you could determine poses and coloration, for one. And in retrospect, you don't need to match that - you just need it to appear obviously pornographic after the blur is applied in order to get past Apple's reviewers. After that, the lucky individual's life is in the hands of the police/prosecutors. (I have to imagine that both real and faked cases will look pretty much like "Your honor/members of the jury, this person's device contained numerous photos matching known CSAM. No, we won't be showing you the pictures. No, the defence can't see them either." Can you imagine a "tough on crime" prosecutor taking the faked case to trial too? Would the police and prosecutors even know it was faked?)
  - floatingatoll 5y ago
    Apple's reviewers are comparing blurred originals to blurred matches. It needs to look like the blurred original associated with the checksum that matched. It is irrelevant whether the blurred match looks pornographic or not.
    - ultrarunner 5y ago
      This has already been demonstrated. https://twitter.com/SarahJamieLewis/status/1428083744280256512 https://twitter.com/SarahJamieLewis/status/14280837442802565...
      - floatingatoll 5y ago
        At which point the image will be handed to the relevant CSAM group unblurred, who will do a visual comparison and find out immediately that it’s not a match, and then reject it without invoking law enforcement.
- ultrarunner 5y ago
  Cops no-knock raid the wrong houses all the time; what makes you think that the procedure you outlined above will be followed without error? Especially against suspected child abusers, from an anti-child-abuse agency? What if the visual check gets accidentally signed off on, or even gets fraudulently marked as positive by a burned out/lazy/competent-but-distracted employee? There are just so many failure modes for this dragnet that all end with innocent people suffering a very difficult process, even if they don't eventually land in prison. I don't think it's unreasonable to not want to be volunteered to participate.
- SXX 5y ago
  Why would attacker need to create anything from scratch? If you want to build a dataset of images with people with specific skin and hair color, body types, etc in very specific poses it's can be quite hard and expensive to do so. Because even photo stocks have limited number of such photos. Unless... Unless you're looking to build a porn dataset and you're don't care about copyright. Porn is industry where exabytes of material are produced and published on internet almost every week. Who will agency come to? To some OnlyFans creators?

treesprite82 5y ago

There are concerns without the images needing to make it all the way through their CSAM process. Apple is a US company with obligations to report certain crimes they become aware of, and NCMEC is a US government-tied organisation that dismissed privacy concerns as "the screeching voices of the minority". Consider if the honeypot images (manipulated to match CSAM hashes) are terrorist recruitment material for example.

int_19h 5y ago

At the point where it is submitted for a human visual check, your privacy has already been violated.

floatingatoll 5y ago
My privacy is violated every time I leave my home. Anyone can take a photo of me and look up the FBI Most Wanted and see if I’m there. If they think someone is me and they’re wrong, they can still summon law enforcement, and I’ll still be mistreated for their poor judgement. Is this just as unacceptable as the CSAM scanning? Should all public photography be banned, in order to reduce the risk of false positive identifications of innocent people as criminals to zero? Or is that an acceptable degree of privacy impingement for the good of society? Is Apple’s implementation an acceptable trade of impingement and risk, for good for society? We do live in a society, and so zero impingement upon privacies is never going to be acceptable (sorry, free/tech/libs). But instead of discussing whether Apple’s approach violates privacy minimally or not in order to get the job done, these discussions here just keep circling the drain of “putting my privacy at risk by any degree is never acceptable”, when that drain is cemented shut by the existence of society and will never lead to a valid outcome.
- int_19h 5y ago
  By leaving your home and going out in public, you are implicitly consenting to being observed. So, no, that's not a violation of your privacy. Apple's implementation is more like a stranger sneaking into your house if they suspect you're up to something bad. That would be equally unacceptable - we have warrants for a reason.