4 ms·

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - th

The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. You'd basically start off with an image known (or very likely) to be cataloged in a CP hash database. Note its NeuralHash. Find a non-CP image that would, after being scaled down or otherwise sanitized, fool an unaccountable and likely disinterested Apple employee into muttering "close enough" while selecting whichever option box it is that causes life ruination. Feed that imagine into an adversarial network until it spits out the desired NeuralHash. Distribute that image to everyone who has ever disagreed with you on the internet, prayed to the wrong god, competed with you in business, voted the wrong way, etc.

jachee 5y ago
… and repeat “on the order of 30+“ times to trigger the reporting threshold, and convince them to save it into their synced-to-iCloud Photos library.
- sennight 5y ago
  So have you never heard of catfishing? Because if you have, then you know it wouldn't be hard to do just what you described - and you're pretending otherwise for some reason.
- Karunamon 5y ago
  ...or just blast the messages to them on WhatsApp or any other tool that automatically dumps incoming images into the photo roll.
cinquemb 5y ago
> The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. Yeah, mentioned something like that here [0]: >> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this. [0] https://news.ycombinator.com/item?id=28177170 https://news.ycombinator.com/item?id=28177170
skygazer 5y ago
my aim was to point out that the above reverenced "image scaling attack" is easily protected against, because it is fragile to alternate scaling methods -- it breaks if you don't use the scaling algorithm the attacker planned for, and there exist secure scaling algorithms that are immune. [0] Since defeating the image scaling attack is trivial, it means that, if it is addressed, the thumbnail will always resemble the full image. With that out of the way, that, obviously, just forecloses this one particular attack, specifically, where you want the thumbnail to appear dramatically different than the full image in order to fool the user that it's an innocent image and the reviewer that it's an illegal image. It's still, never-the-less, possible to have a confusing thumbnail -- perhaps an adult porn image engineered to have a CSAM hash collision will be enough to convince a beleaguered or overeager reviewer to pull the trigger. The "Image Scaling Attack" is neither sufficient or necessary. (However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review, so would never be shown to a human reviewer: "as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash" [1]) [0] Understanding and Preventing Image-Scaling Attacks in Machine Learning https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...
- sennight 5y ago
  > However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review... Uh, on what timescale? If you mean "tomorrow" then sure, if you mean "for years" - then no. They're relying on the second perceptual hashing algorithm to remain a secret, which is insanely foolish. Just based on what I know about these CP hashlists and the laziness of programmers, I feel pretty confident that it is either an algorithm trained on the thumbnails themselves (which would be laughably bad) or it was a prior attempt that got replaced by what is now deployed on the users' hardware. Why would I think that? Because it would have been the only other thing on hand for the necessary step of generating the hash black list. So they're stuck with at least one of those forever - and will have a very limited range of potential responses to the massive infosec spotlight picking them apart... unless they want to recatalog every bit of CP all over again.
FranksTV 5y ago
Why not just put actual child porn on their phone if you have access and want to screw with them? And if you can do that, why wait for apple to implement this? Why not just put child porn on their phone and then call the police?
- cinquemb 5y ago
  > Why not just put child porn on their phone and then call the police? That does not scale. > … why wait for apple to implement this? Enables scaling of what the above commentator discusses.

jachee 5y ago

… and repeat “on the order of 30+“ times to trigger the reporting threshold, and convince them to save it into their synced-to-iCloud Photos library.

sennight 5y ago
So have you never heard of catfishing? Because if you have, then you know it wouldn't be hard to do just what you described - and you're pretending otherwise for some reason.
Karunamon 5y ago
...or just blast the messages to them on WhatsApp or any other tool that automatically dumps incoming images into the photo roll.

cinquemb 5y ago

> The objective of being mindful of the thumbnail is to fool the human reviewer responsible for alerting the police to your target's need for a good swatting - the algorithm has already flagged the image by the time it is presented as a thumbnail during review. Yeah, mentioned something like that here [0]: >> And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this. [0] https://news.ycombinator.com/item?id=28177170 https://news.ycombinator.com/item?id=28177170

skygazer 5y ago

my aim was to point out that the above reverenced "image scaling attack" is easily protected against, because it is fragile to alternate scaling methods -- it breaks if you don't use the scaling algorithm the attacker planned for, and there exist secure scaling algorithms that are immune. [0] Since defeating the image scaling attack is trivial, it means that, if it is addressed, the thumbnail will always resemble the full image. With that out of the way, that, obviously, just forecloses this one particular attack, specifically, where you want the thumbnail to appear dramatically different than the full image in order to fool the user that it's an innocent image and the reviewer that it's an illegal image. It's still, never-the-less, possible to have a confusing thumbnail -- perhaps an adult porn image engineered to have a CSAM hash collision will be enough to convince a beleaguered or overeager reviewer to pull the trigger. The "Image Scaling Attack" is neither sufficient or necessary. (However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review, so would never be shown to a human reviewer: "as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash" [1]) [0] Understanding and Preventing Image-Scaling Attacks in Machine Learning https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf https://www.sec.cs.tu-bs.de/pubs/2020-sec.pdf [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model...

sennight 5y ago
> However, that confusing image would almost certainly not also fool Apple's unspecified secondary server-side hashing algorithm, as referenced on page 13 of Apple's Security Threat Model Review... Uh, on what timescale? If you mean "tomorrow" then sure, if you mean "for years" - then no. They're relying on the second perceptual hashing algorithm to remain a secret, which is insanely foolish. Just based on what I know about these CP hashlists and the laziness of programmers, I feel pretty confident that it is either an algorithm trained on the thumbnails themselves (which would be laughably bad) or it was a prior attempt that got replaced by what is now deployed on the users' hardware. Why would I think that? Because it would have been the only other thing on hand for the necessary step of generating the hash black list. So they're stuck with at least one of those forever - and will have a very limited range of potential responses to the massive infosec spotlight picking them apart... unless they want to recatalog every bit of CP all over again.

FranksTV 5y ago

Why not just put actual child porn on their phone if you have access and want to screw with them? And if you can do that, why wait for apple to implement this? Why not just put child porn on their phone and then call the police?

cinquemb 5y ago
> Why not just put child porn on their phone and then call the police? That does not scale. > … why wait for apple to implement this? Enables scaling of what the above commentator discusses.