5 ms·

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a l

It's entirely possible to alter an image such that its raw form looks different from its scaled form [0]. A government or just well resourced group can take a legitimate CSAM image and modify it such that when scaled for use in the perceptual algorithm(s) it changes to be some politically sensitive image. Upon review it'll look like CSAM so off it goes to reporting agencies. Because the perceptual hash algorithms are presented as black boxes the image they perceive isn't audited or reviewed. There's zero recognition of this weakness by Apple or NCMEC (and their equivalents). For the system to even begin to be trustworthy all content would need to be reviewed raw and scaled-as-fed-into-the-algorithm. [0] https://bdtechtalks.com/2020/08/03/machine-learning-adversarial-image-scaling/ https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

skygazer 5y ago
This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.
- sennight 5y ago
  The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination. Feed that imagine into an adversarial network until it spits out the desired NeuralHash. Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.
  - jachee 5y ago
    … and repeat “on the order of 30+“ times to trigger the reporting threshold, and convince them to save it into their synced-to-iCloud Photos library.
    - sennight 5y ago
      So have you never heard of catfishing? Because if you have, then you know it wouldn't be hard to do just what you described - and you're pretending otherwise for some reason.
    - Karunamon 5y ago
      ...or just blast the messages to them on WhatsApp or any other tool that automatically dumps incoming images into the photo roll.
  - cinquemb 5y ago
    > The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. Yeah, mentioned something like that here [0]: >> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this. [0] https://news.ycombinator.com/item?id=28177170 https://news.ycombinator.com/item?id=28177170
  - skygazer 5y ago
    my aim was to point out that the above reverenced "image scaling attack" is easily protected against, because it is fragile to alternate scaling methods -- it breaks if you don't use the scaling algorithm the attacker planned for, and there exist secure scaling algorithms that are immune. [0] Since defeating the image scaling attack is trivial, it means that, if it is addressed, the thumbnail will always resemble the full image. With that out of the way, that, obviously, just forecloses this one particular attack, specifically, where you want the thumbnail to appear dramatically different than the full image in order to fool the user that it's an innocent image and the reviewer that it's an illegal image. It's still, never-the-less, possible to have a confusing thumbnail -- perhaps an adult porn image engineered to have a CSAM hash collision will be enough to convince a beleaguered or overeager reviewer to pull the trigger. The "Image Scaling Attack" is neither sufficient or necessary. (However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review, so would never be shown to a human reviewer: "as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash" [1]) [0] Understanding and Preventing Image-Scaling Attacks in Machine Learning https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
- orivej 5y ago
  The attack relies on the fact that when downscaling by a large factor, the tested downscalers (except Pillow in non-nearest neighmode mode, and all of them in area averaging mode) ignore most of the pixels of the original image and compute the result based on the select few which are the same in all modes, making the result look nearly the same regardless of the mode.
  - skygazer 5y ago
    Thanks for that reference to Pillow. I presume it's from "Understanding and Preventing Image-Scaling Attacks in Machine Learning" [0] which mentions secure scaling algorithms immune to the attack. I wish I could mention this in the grand parent, but the editing window closed. [0] https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf
sandos 5y ago
That has to be scaling where you dont properly get rid of all the high-frequency bits of information before scaling, no? I have a hard time seeing this happen with a "perfect" scaling algoritm in the spatial domain.
BizarroLand 5y ago
If you know the hash of an approved image, say of a puppy, a malicious person could edit a CP image to have the same hash as the puppy, right? Wouldn't that mean that the system can be poisoned? How would they deal with that?
- giantrobot 5y ago
  Just a correction for you, there's not a list of approved images. The CSAM database are a list of illegal (unapproved if you will) images. Other than that, yes it's possible to add noise to an image so a perceptual algorithm misidentifies it. I described the false positive case but it can also be used for false negatives. Someone can apply noise to a legit CSAM image (in the NCMEC database) so Apple's system fails to identify it. The false positive case is scary because if it happens to you your life is ruined. The false negative case just means people have CSAM and don't get found by the system. I'm much more concerned about the false positive case.
  - saithound 5y ago
    Keep in mind that there are multiple straight paths from the false negative case to the false positive case. I'll give you one examples: pedos can and will use the collider to produce large batches of CSAM that collide with perfectly legitimate images (e.g. common iPhone wallpapers). They literally have nothing to lose by doing this. Eventually, these photos will make their way into the NCMEC database, and produce a large number of false positives. This will also make the other attacks discussed here easier to execute (e.g. by lowering the human review threshold, since everybody will start with a few strikes).

skygazer 5y ago

This attack does seem easily defeated, even naively, by downscaling by three different means (bicubic, nearest neighbor, Lanczos, etc.) and rejecting the downscale that most differs from the other two, since the attack is tailored to a specific downscaling algorithm -- the attack seems to only be effective against systems that make no effort at all to safeguard against it. Granted, Apple makes no mention of any safeguard, but it would be trivial in principle to protect against, and is not an unavoidable failing.

sennight 5y ago
The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination. Feed that imagine into an adversarial network until it spits out the desired NeuralHash. Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.
- jachee 5y ago
  … and repeat “on the order of 30+“ times to trigger the reporting threshold, and convince them to save it into their synced-to-iCloud Photos library.
  - sennight 5y ago
    So have you never heard of catfishing? Because if you have, then you know it wouldn't be hard to do just what you described - and you're pretending otherwise for some reason.
  - Karunamon 5y ago
    ...or just blast the messages to them on WhatsApp or any other tool that automatically dumps incoming images into the photo roll.
- cinquemb 5y ago
  > The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. Yeah, mentioned something like that here [0]: >> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this. [0] https://news.ycombinator.com/item?id=28177170 https://news.ycombinator.com/item?id=28177170
- skygazer 5y ago
  my aim was to point out that the above reverenced "image scaling attack" is easily protected against, because it is fragile to alternate scaling methods -- it breaks if you don't use the scaling algorithm the attacker planned for, and there exist secure scaling algorithms that are immune. [0] Since defeating the image scaling attack is trivial, it means that, if it is addressed, the thumbnail will always resemble the full image. With that out of the way, that, obviously, just forecloses this one particular attack, specifically, where you want the thumbnail to appear dramatically different than the full image in order to fool the user that it's an innocent image and the reviewer that it's an illegal image. It's still, never-the-less, possible to have a confusing thumbnail -- perhaps an adult porn image engineered to have a CSAM hash collision will be enough to convince a beleaguered or overeager reviewer to pull the trigger. The "Image Scaling Attack" is neither sufficient or necessary. (However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review, so would never be shown to a human reviewer: "as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash" [1]) [0] Understanding and Preventing Image-Scaling Attacks in Machine Learning https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
orivej 5y ago
The attack relies on the fact that when downscaling by a large factor, the tested downscalers (except Pillow in non-nearest neighmode mode, and all of them in area averaging mode) ignore most of the pixels of the original image and compute the result based on the select few which are the same in all modes, making the result look nearly the same regardless of the mode.
- skygazer 5y ago
  Thanks for that reference to Pillow. I presume it's from "Understanding and Preventing Image-Scaling Attacks in Machine Learning" [0] which mentions secure scaling algorithms immune to the attack. I wish I could mention this in the grand parent, but the editing window closed. [0] https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf

sandos 5y ago

That has to be scaling where you dont properly get rid of all the high-frequency bits of information before scaling, no? I have a hard time seeing this happen with a "perfect" scaling algoritm in the spatial domain.

BizarroLand 5y ago

If you know the hash of an approved image, say of a puppy, a malicious person could edit a CP image to have the same hash as the puppy, right? Wouldn't that mean that the system can be poisoned? How would they deal with that?

giantrobot 5y ago
Just a correction for you, there's not a list of approved images. The CSAM database are a list of illegal (unapproved if you will) images. Other than that, yes it's possible to add noise to an image so a perceptual algorithm misidentifies it. I described the false positive case but it can also be used for false negatives. Someone can apply noise to a legit CSAM image (in the NCMEC database) so Apple's system fails to identify it. The false positive case is scary because if it happens to you your life is ruined. The false negative case just means people have CSAM and don't get found by the system. I'm much more concerned about the false positive case.
- saithound 5y ago
  Keep in mind that there are multiple straight paths from the false negative case to the false positive case. I'll give you one examples: pedos can and will use the collider to produce large batches of CSAM that collide with perfectly legitimate images (e.g. common iPhone wallpapers). They literally have nothing to lose by doing this. Eventually, these photos will make their way into the NCMEC database, and produce a large number of false positives. This will also make the other attacks discussed here easier to execute (e.g. by lowering the human review threshold, since everybody will start with a few strikes).