4 ms·

"The system could only match "exact fingerprints" of specific known child sexual abuse images, he said." This disinfo really angers me. That is the exact oppo

by throw7 5y ago

"The system could only match "exact fingerprints" of specific known child sexual abuse images, he said." This disinfo really angers me. That is the exact opposite of what I've read up till now. People talking about "NeuralHash" and being able to detect if the image is cropped/edited/"similar". SO what is the truth?

JohnCurran 5y ago
That "exact fingerprint" is, in my opinion, intentionally confusing. This DaringFireball[0] article states the goal of the system is to "generate the same fingerprint identifier if the same image is cropped, resized, or even changed from color to grayscale." So while the fingerprint may be "exact", it's still capable of detecting images which have been altered in some way [0] https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia...
patrickthebold 5y ago
If I had to guess, cropping and other transformations result in the same (exact) fingerprint. So different images but same fingerprints. Of course, that's just a nasty way to imply that the images match exactly.
gizdan 5y ago
The truth is they're rephrasing what was already known. They're going to match match finger prints of pictures against a database. Every picture. This was widely report. What confusion they're referring to I don't know, because they're saying exactly what has been reported.
- notJim 5y ago
  > Every picture This is the confusion, it's only photos being uploaded to iCloud.
  - salamandersauce 5y ago
    So almost every picture. Isn't it the default to upload all photos to iCloud on iOS devices? Doesn't it even helpfully remove photos that aren't used as much to make room for new ones?
    - kemayo 5y ago
      I can't find an answer about whether it's the default nowadays. You certainly used to have to turn it on -- e.g. the Apple support page on the feature tells you how to do so: https://support.apple.com/en-us/HT204264 https://support.apple.com/en-us/HT204264 That said, the argument that many people in these threads are making is that they say it's reasonable to scan photos that are uploaded once they're on Apple's servers, they just don't want them scanned while they're still on their phones. In either case, the same photos will be scanned -- ones which are in the process of being uploaded to iCloud -- the disagreement is just about exactly when in said process it's okay to do so. Which seems like a pretty fine distinction to me?
btown 5y ago
It's all on pages 4 and 5 of https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... > The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, and images that are different from one another result in different hashes. For example, an image that has been slightly cropped or resized should be considered identical to its original and have the same hash. The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements Just like a human fingerprint is a lower-dimensional representation of all the atoms in your body that's invariant to how old you are or the exact stance you're in when you're fingerprinted... technically Federighi is being accurate about the "exact fingerprint" part. The thing that has me and others concerned isn't necessarily the hash algorithm per se, but rather: how can Apple promise to the world that the data source for "specific known child sexual abuse images" will actually be just that over time? There are two attacks of note: (1) a sophisticated actor compromising the hash list handoff from NCMEC to Apple to insert hashes of non-CSAM material, which is something Apple cannot independently verify as it does not have access to the raw images, which at minimum could be a denial-of-service attack causing e.g. journalists' or dissidents' accounts to be frozen temporarily by Apple's systems pending appeal (2) Apple no longer being able to have a "we don't think we can do this technically due to our encryption" leg to stand on when asked by foreign governments "hey we have a list of hashes, just create a CSAM-like system for us" That Apple must have considered these possibilities and built this system anyways is a tremendously significant breach of trust.
- mzs 5y ago
  Thanks, here's how one system works, calculates dozens of floating point moments: http://www.fmwconcepts.com/misc_tests/perceptual_hash_test_results_510/index.html http://www.fmwconcepts.com/misc_tests/perceptual_hash_test_r...
LeifCarrotson 5y ago
He carefully avoided saying that the image itself is the same. The exact fingerprint is the same, yes, but the fingerprint is just a hash of the actual image. Disinformation indeed! The whole point of the system is that you get a matching hash after mirroring/rotating/distorting/cropping/compressing/transforming/watermarking the source image. The system would be pretty useless if it couldn't match an image after someone, say, added a watermark. And if the algorithm was public, it would be easy to bypass. The concern, of course, is that all of this many-to-one hashing might also cause another unrelated image to generate the same fingerprint, and thereby throw an innocent person to an unyielding blankface bureaucracy who believes their black-box system without question.
- 734129837261 5y ago
  It simply means that they can have whatever the hell kind of method they use to identify specific images, and the scary part is: there IS an error-margin built-in because otherwise, as you said, this tech would be pretty useless. "Find all images and tag them if they look like this fingerprint" doesn't mean that. It means: "Find all images and tag them if they look 80% like this fingerprint". Which also means that it will allow governments to upload photographs of people's faces and say: "Tag anyone who looks like this". Worse, this will allow China to track down more Uyghurs, find people based on guides in the form of images that are spread around to stay safe from the Chinese government, and countries like Saudi Arabia can start looking for phones with a significant amount of atheist-related images, tracking down atheists, and killing them. Because that's what that country does.
  - izend 5y ago
    The CCP has had access to iCloud in China for multiple years... https://www.reuters.com/article/us-china-apple-icloud-insight-idUSKCN1G8060 https://www.reuters.com/article/us-china-apple-icloud-insigh...
- intricatedetail 5y ago
  These perceptual hashes do have high number of false positives. That's why they employ AI to discard images that don't have certain features from the pool to minimise the risk. But that method in general without actual human checking manually is a recipe for disaster.
ddlutz 5y ago
And we all know software never has bugs. Somebody is going to get arrested over this feature for some benign photo one day, I guarantee it.
- nomel 5y ago
  How so? It would require passing the threshold to get human review, so actual material + false flags > threshold. This should probably result in the person getting in trouble. The case of false flags > threshold should not result in an any trouble since it would then go through human review.
totetsu 5y ago
Just wait till you download that meme image, to upload to your reaction meme folder on icloud, that some troll has kept the background of some csam image and edited meme text over the bits that might have made you aware of its origins. will that match?
- laurent92 5y ago
  Who cares, the NCMEC database is certainly full of unreviewed material, given even their employees can’t automatically have access to it. For any dystopian state, the goal is to have as many false positives as possible in the NCMEC database, to be able to legitimately have your photos uploaded to their headquarters.
foobiekr 5y ago
An exact match of a perceptual hash is basically deliberately misleading. The entire point of a perceptual hash is that there are an almost unlimited number of images which it will match "exactly." But hey, I'm just one of the screeching voices of the minority.
- intricatedetail 5y ago
  It will also match completely different images, that why there is "neural" bit to discard images that e.g. don't have nudity from the pool of matches.
  - pseudalopex 5y ago
    The neural bit doesn't do that.[1] Maybe you got it mixed up with the iMessage nudity detection. [1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf#page=5 https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
    - intricatedetail 5y ago
      Did you read it? They use neural network to discard false positives, because perceptual hash alone is not reliable. It's pretty much the same concept I described.
      - pseudalopex 5y ago
        Please quote what you think supports your claim. "Indeed, Neural-Hash knows nothing at all about CSAM images. It is an algorithm designed to answer whether one image is really the same image as another, even if some image-altering transformations have been applied (like transcoding, resizing, and cropping)."[1] [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf#page=10 https://www.apple.com/child-safety/pdf/Security_Threat_Model...
        dang 5y ago
elliekelly 5y ago
Does it make a difference? My iPhone shouldn’t do anything to or with my photos unless and until I direct it to. Scanning, hashing, whatevering — Apple doesn’t get to decide to do any of it. I do. And only I do.
blintz 5y ago
The simple summary is: NeuralHash is not a cryptographic hash function. It's a private neural network trained on some images. We have no guarantees of its difficulty to reverse, find collisions for, etc. The naming of it as a 'hash' has confused people (John Gruber's post comes to mind) into thinking this is a cryptographic hash. It simply is not.

by throw7 5y ago

JohnCurran 5y ago

That "exact fingerprint" is, in my opinion, intentionally confusing. This DaringFireball[0] article states the goal of the system is to "generate the same fingerprint identifier if the same image is cropped, resized, or even changed from color to grayscale." So while the fingerprint may be "exact", it's still capable of detecting images which have been altered in some way [0] https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope https://daringfireball.net/2021/08/apple_child_safety_initia...

patrickthebold 5y ago

If I had to guess, cropping and other transformations result in the same (exact) fingerprint. So different images but same fingerprints. Of course, that's just a nasty way to imply that the images match exactly.

gizdan 5y ago

The truth is they're rephrasing what was already known. They're going to match match finger prints of pictures against a database. Every picture. This was widely report. What confusion they're referring to I don't know, because they're saying exactly what has been reported.

notJim 5y ago
> Every picture This is the confusion, it's only photos being uploaded to iCloud.
- salamandersauce 5y ago
  So almost every picture. Isn't it the default to upload all photos to iCloud on iOS devices? Doesn't it even helpfully remove photos that aren't used as much to make room for new ones?
  - kemayo 5y ago
    I can't find an answer about whether it's the default nowadays. You certainly used to have to turn it on -- e.g. the Apple support page on the feature tells you how to do so: https://support.apple.com/en-us/HT204264 https://support.apple.com/en-us/HT204264 That said, the argument that many people in these threads are making is that they say it's reasonable to scan photos that are uploaded once they're on Apple's servers, they just don't want them scanned while they're still on their phones. In either case, the same photos will be scanned -- ones which are in the process of being uploaded to iCloud -- the disagreement is just about exactly when in said process it's okay to do so. Which seems like a pretty fine distinction to me?

btown 5y ago

It's all on pages 4 and 5 of https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... > The main purpose of the hash is to ensure that identical and visually similar images result in the same hash, and images that are different from one another result in different hashes. For example, an image that has been slightly cropped or resized should be considered identical to its original and have the same hash. The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements Just like a human fingerprint is a lower-dimensional representation of all the atoms in your body that's invariant to how old you are or the exact stance you're in when you're fingerprinted... technically Federighi is being accurate about the "exact fingerprint" part. The thing that has me and others concerned isn't necessarily the hash algorithm per se, but rather: how can Apple promise to the world that the data source for "specific known child sexual abuse images" will actually be just that over time? There are two attacks of note: (1) a sophisticated actor compromising the hash list handoff from NCMEC to Apple to insert hashes of non-CSAM material, which is something Apple cannot independently verify as it does not have access to the raw images, which at minimum could be a denial-of-service attack causing e.g. journalists' or dissidents' accounts to be frozen temporarily by Apple's systems pending appeal (2) Apple no longer being able to have a "we don't think we can do this technically due to our encryption" leg to stand on when asked by foreign governments "hey we have a list of hashes, just create a CSAM-like system for us" That Apple must have considered these possibilities and built this system anyways is a tremendously significant breach of trust.

mzs 5y ago
Thanks, here's how one system works, calculates dozens of floating point moments: http://www.fmwconcepts.com/misc_tests/perceptual_hash_test_results_510/index.html http://www.fmwconcepts.com/misc_tests/perceptual_hash_test_r...

LeifCarrotson 5y ago

He carefully avoided saying that the image itself is the same. The exact fingerprint is the same, yes, but the fingerprint is just a hash of the actual image. Disinformation indeed! The whole point of the system is that you get a matching hash after mirroring/rotating/distorting/cropping/compressing/transforming/watermarking the source image. The system would be pretty useless if it couldn't match an image after someone, say, added a watermark. And if the algorithm was public, it would be easy to bypass. The concern, of course, is that all of this many-to-one hashing might also cause another unrelated image to generate the same fingerprint, and thereby throw an innocent person to an unyielding blankface bureaucracy who believes their black-box system without question.

734129837261 5y ago
It simply means that they can have whatever the hell kind of method they use to identify specific images, and the scary part is: there IS an error-margin built-in because otherwise, as you said, this tech would be pretty useless. "Find all images and tag them if they look like this fingerprint" doesn't mean that. It means: "Find all images and tag them if they look 80% like this fingerprint". Which also means that it will allow governments to upload photographs of people's faces and say: "Tag anyone who looks like this". Worse, this will allow China to track down more Uyghurs, find people based on guides in the form of images that are spread around to stay safe from the Chinese government, and countries like Saudi Arabia can start looking for phones with a significant amount of atheist-related images, tracking down atheists, and killing them. Because that's what that country does.
- izend 5y ago
  The CCP has had access to iCloud in China for multiple years... https://www.reuters.com/article/us-china-apple-icloud-insight-idUSKCN1G8060 https://www.reuters.com/article/us-china-apple-icloud-insigh...
intricatedetail 5y ago
These perceptual hashes do have high number of false positives. That's why they employ AI to discard images that don't have certain features from the pool to minimise the risk. But that method in general without actual human checking manually is a recipe for disaster.

ddlutz 5y ago

And we all know software never has bugs. Somebody is going to get arrested over this feature for some benign photo one day, I guarantee it.

nomel 5y ago
How so? It would require passing the threshold to get human review, so actual material + false flags > threshold. This should probably result in the person getting in trouble. The case of false flags > threshold should not result in an any trouble since it would then go through human review.

totetsu 5y ago

Just wait till you download that meme image, to upload to your reaction meme folder on icloud, that some troll has kept the background of some csam image and edited meme text over the bits that might have made you aware of its origins. will that match?

laurent92 5y ago
Who cares, the NCMEC database is certainly full of unreviewed material, given even their employees can’t automatically have access to it. For any dystopian state, the goal is to have as many false positives as possible in the NCMEC database, to be able to legitimately have your photos uploaded to their headquarters.

foobiekr 5y ago

An exact match of a perceptual hash is basically deliberately misleading. The entire point of a perceptual hash is that there are an almost unlimited number of images which it will match "exactly." But hey, I'm just one of the screeching voices of the minority.

intricatedetail 5y ago
It will also match completely different images, that why there is "neural" bit to discard images that e.g. don't have nudity from the pool of matches.
- pseudalopex 5y ago
  The neural bit doesn't do that.[1] Maybe you got it mixed up with the iMessage nudity detection. [1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf#page=5 https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - intricatedetail 5y ago
    Did you read it? They use neural network to discard false positives, because perceptual hash alone is not reliable. It's pretty much the same concept I described.
    - pseudalopex 5y ago
      Please quote what you think supports your claim. "Indeed, Neural-Hash knows nothing at all about CSAM images. It is an algorithm designed to answer whether one image is really the same image as another, even if some image-altering transformations have been applied (like transcoding, resizing, and cropping)."[1] [1] https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf#page=10 https://www.apple.com/child-safety/pdf/Security_Threat_Model...
      - dang 5y ago

elliekelly 5y ago

Does it make a difference? My iPhone shouldn’t do anything to or with my photos unless and until I direct it to. Scanning, hashing, whatevering — Apple doesn’t get to decide to do any of it. I do. And only I do.

blintz 5y ago

The simple summary is: NeuralHash is not a cryptographic hash function. It's a private neural network trained on some images. We have no guarantees of its difficulty to reverse, find collisions for, etc. The naming of it as a 'hash' has confused people (John Gruber's post comes to mind) into thinking this is a cryptographic hash. It simply is not.