6 ms·

I don't think Apple has ever claimed that NeuralHash was hardened against intentional false positives. And when properly considered as just one component of the

I don't think Apple has ever claimed that NeuralHash was hardened against intentional false positives. And when properly considered as just one component of their CSAM scanning architecture, NeuralHash being hardened against intentional false positives isn't necessary for it to serve its purpose.

joe_the_user 5y ago
Uh, the gp points both intentional false positives and intentional false negatives (IE, anyone intentionally keeping CSAM can easily hide it while anyone can create an image that triggers the hash and gets human to scan your image and etc).
- simondotau 5y ago
  Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly. Intentional false positives require knowledge of a hash. And even that isn't enough to get a human to scan your image. You would need to acquire a library of A1 CSAM images/hashes. You would need to generate images which match both NeuralHash and the unspecified secondary hash algorithm which runs on Apple's cloud servers. And then you'd need to repeat this trick a sufficient number of times to trigger human review, at which point a few people (probably some minions in a developing country) get to look at the images for 1 second before pressing the "not CSAM" button and having the images sent to a technical team in Cupertino for analysis.
  - ClumsyPilot 5y ago
    If both false positives and false negatives are irrelevant, then the whole system and argument is irrelevant.
    - simondotau 5y ago
      Huh? False negatives and false positives are irrelevant with respect to the utility of the overall system. It means they don't render the system irrelevant. Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery. Intentional false positives are irrelevant because there are, depending how you count them, between two and six other layers which would need to be thwarted or exploited in order for intentional false positives in NeuralHash to result in any adverse outcome.
      - DeathArrow 5y ago
        >Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery. So if the purpose of the system is not to catch persons dealing with CP, what is the purpose? Breaking user's privacy? We have a history of evil intentions masked by fighting against CP, drugs, money laundry and crime in general.
        emodendroket 5y ago
        There is pretty much no enforcement mechanism against any crime which we can expect to catch everyone. False positives get bounced out by the safeguards, false negatives wouldn't have been caught in the first place... that still presumably leaves a number of true positives who wouldn't have otherwise been caught.
        kalleboo 5y ago
        Facebook is catching millions of CSAM images being directly uploaded by people to Facebook/Messenger of all places. There are still plenty of complete idiots trading CSAM that one could see value in apprehending.
    - maratc 5y ago
      To the contrary. A system that detects Covid-19 in infected people and doesn't detect it in healthy people, where the errors of both types (false positives and false negatives) are irrelevant, is not only relevant, it's a very well designed system.
  - joe_the_user 5y ago
    Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly. So, you're accepting that Apple's approach is useless for anyone who's "determined" but you still seem to be arguing there's some merit in this complex "architecture" of surveillance.
    - simondotau 5y ago
      I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already. And if so, it would be no small irony that the noisy criticism would have ensured that Apple was heard loud and clear by the CSAM community.
      - joe_the_user 5y ago
        I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already. So the idea is that mass surveillance is great even if it's effect is scaring criminals while it actually only catches people at random? We've arrived at the opposite of the common reprise of law. No longer "It is Better that Ten Guilty Persons Go Free Than that One Innocent Person be Convicted" but "It is better that Ten Innocent Persons Be Dragooned Than that One Guilty Person Go Free"
        simondotau 5y ago
        All I'm saying is that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future. It's a tough line for them to walk. Personally I don't see any substantial moral difference to what Apple was planning versus what Google and Facebook already do, aside for the somewhat icky aspect that some initial CPU cycles are occurring on hardware I own. I'm equally concerned that a purported need to scan for CSAM could be used by legislators as a way to convince the population that the Government needs more encryption backdoors. Personally I'd rather have my photos inconsequentially hashed than have encryption legislatively broken.
  - chmod775 5y ago
    > Intentional false positives require knowledge of a hash. The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known. You would have to hope that the secondary system is robust enough to not fall victim to the same manipulated image. And it really is on Apple to first prove it isn't.
    - jdlshore 5y ago
      No, it is not. The hash is computed on the client, but hash comparison happens on the server, using some cryptographic magic that only reveals a “visual derivative” if enough images match. It’s fairly complex and I don’t recall the details, but the library of “bad” hashes is not accessible.
      - tsimionescu 5y ago
        No, the entire premise of NeuralHash is that no information about your pictures leaves your system unless it has been flagged as 'potentially CSAM'. NeuralHash itself is basically a DB of CSAM hashes encoded as an artificial neural network. In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash. Edit: I am almost entirely wrong, see the comment reply from krisoft. Apologies!
        krisoft 5y ago
        > In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash. No. You are wrong. They do hash each image and they do upload the hash derived safety voucher always. Here is the Apple technical paper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... There is a fairly complicated cryptographic algorithm which, according to apple, lets them open the safety voucher only if there is a match with the database. The user devices only have a cryptographicaly blinded version of the hash database which they can’t turn back to real hashes without apple’s private key. This means that the plain neuralhashes are not available to the end users.
  - kf6nux 5y ago
    The other reason why false negatives are mostly irrelevant is because the hash DB can be updated to include them, so the risk of storing them in iCloud remains the same.
    - nullc 5y ago
      That doesn't fly. The encrypted list is stored on the devices for security and performance reasons: otherwise it would be too easy to ship people customized lists with arbitrary images. You can't just rapidly revise the list without undermining the privacy protection here. Also the space of modifications is gigantic, for any given image there should be millions of differently hash variations which can be reached without major visual damage. (at least for neuralhash the figure is billions)... so you can't just stick all of them in the database. You could stick widely circulating variations in the database, however... sure.
      - kf6nux 5y ago
        I'm honestly not sure what you're refuting here. Are you saying the hash DB can't be updated or that already uploaded photos won't be compared to hash updates?
        nullc 5y ago
        The hashdb is part of the operating system and only gets updated with OS updates. It can be updated-- at some cost. But if it is very frequently updated then the protection against receiving a targeted list is lost. To state it another way: Users ask "Why won't the government order apple just gives me a list of the hash of every political image on the internet in order to expose my files?" and the answer is that the database is part of the OS, common to everyone, updated infrequently, so it can't be used in targeted attacks. That protection is lost if the database is frequently updated. It's also true that already uploaded photos won't be rescanned, but that wasn't the point I was trying to make.
nullc 5y ago
The article and kristjansson's post are about intentional false negatives. My comment was pointing out that the well known fact that the neuralhash is highly vulnerable to preimage attacks (intentional false positives) also means that is is highly vulnerable to intentional false negatives. Changing the 96-bit hash to a specific value while minimizing the visible image change is a LOT harder than just changing any single bit, but the harder task has been amply demonstrated. Neuralhash is one component, yes, but it is a limiting component. If a slightly tweaked image has a different NeuralHash it will never be detected by their scheme.
- simondotau 5y ago
  I don't think Apple is concerned about the <0.1% of CSAM aficionados who read about Apple's plans to scan for CSAM and thinks "hey, let me blindly trust some tool I found on the internet to slightly mangle my highly illegal images so that I can continue co-mingling them with personal photos in iCloud, accessible to anyone who picks up my phone while unlocked. Or any other device synchronised to the same iCloud account." The App Store is presumably full of easy to use apps which can store photo galleries in a separately encrypted vault. This is where people will go after being shooed away from iCloud, not image manglers.
chmod775 5y ago
> being hardened against intentional false positives isn't necessary for it to serve its purpose. Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it. Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM. Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives. I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion). That won't go over well once the first people share such an experience.
- simondotau 5y ago
  This is false. 1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers. 2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review. 3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced. 4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
  - tsimionescu 5y ago
    Your argument works just as well if for any x, NeuralHash(x)=1. The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.
ddevault 5y ago
Say I'm a cop who wants to get into your phone but can't get a warrant. I happen to have an image on my (Android) phone which looks benign to a human, but triggers Apple's CSAM detectors when scanned. I text it to you, the CSAM detector goes off, and suddenly I have probable cause. Cops plant illegal drugs on suspects all the time. Why not this?
- pfortuny 5y ago
  Imagine a politician receiving dozens of photos which trigger this CSAM algorithm because he has iCloud photos as defaul, and then somehow someone gets wind of the fact… BANG: one less. Try to explain to the public that it is only the first line of defense against child sex and that it does not imply anything at all.
- simondotau 5y ago
  Because it wouldn't work. 1. You would need at least 30 such images; 2. You'd have to somehow convince your target to import all 30 images into their photo gallery; 3. The chances of the images accidentally passing the human review step is infinitesimal; 4. Unlike a bag of drugs which actually do contain drugs, these fake images would be like police finding a bag of sugar with "FAKE COCAINE FOR MANUFACTURING PROBABLE CAUSE" written on it.

joe_the_user 5y ago

Uh, the gp points both intentional false positives and intentional false negatives (IE, anyone intentionally keeping CSAM can easily hide it while anyone can create an image that triggers the hash and gets human to scan your image and etc).

simondotau 5y ago
Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly. Intentional false positives require knowledge of a hash. And even that isn't enough to get a human to scan your image. You would need to acquire a library of A1 CSAM images/hashes. You would need to generate images which match both NeuralHash and the unspecified secondary hash algorithm which runs on Apple's cloud servers. And then you'd need to repeat this trick a sufficient number of times to trigger human review, at which point a few people (probably some minions in a developing country) get to look at the images for 1 second before pressing the "not CSAM" button and having the images sent to a technical team in Cupertino for analysis.
- ClumsyPilot 5y ago
  If both false positives and false negatives are irrelevant, then the whole system and argument is irrelevant.
  - simondotau 5y ago
    Huh? False negatives and false positives are irrelevant with respect to the utility of the overall system. It means they don't render the system irrelevant. Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery. Intentional false positives are irrelevant because there are, depending how you count them, between two and six other layers which would need to be thwarted or exploited in order for intentional false positives in NeuralHash to result in any adverse outcome.
    - DeathArrow 5y ago
      >Intentional false negatives are irrelevant because catching 100% of CSAM was never stated as a goal of the system. And because it's just not realistic to expect to see any real-world use by CSAM aficionados. Anyone savvy enough to understand the principles behind hashing (and trust some command line tool downloaded from Github) is probably smart enough to have better places to store their child porn than an iCloud gallery. So if the purpose of the system is not to catch persons dealing with CP, what is the purpose? Breaking user's privacy? We have a history of evil intentions masked by fighting against CP, drugs, money laundry and crime in general.
      - emodendroket 5y ago
        There is pretty much no enforcement mechanism against any crime which we can expect to catch everyone. False positives get bounced out by the safeguards, false negatives wouldn't have been caught in the first place... that still presumably leaves a number of true positives who wouldn't have otherwise been caught.
      - kalleboo 5y ago
        Facebook is catching millions of CSAM images being directly uploaded by people to Facebook/Messenger of all places. There are still plenty of complete idiots trading CSAM that one could see value in apprehending.
  - maratc 5y ago
    To the contrary. A system that detects Covid-19 in infected people and doesn't detect it in healthy people, where the errors of both types (false positives and false negatives) are irrelevant, is not only relevant, it's a very well designed system.
- joe_the_user 5y ago
  Intentional false negatives are mostly irrelevant. If you want to store galleries of illegal material on your phone, there are much better ways to do it which don't involve them being visually mangled, even if only slightly. So, you're accepting that Apple's approach is useless for anyone who's "determined" but you still seem to be arguing there's some merit in this complex "architecture" of surveillance.
  - simondotau 5y ago
    I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already. And if so, it would be no small irony that the noisy criticism would have ensured that Apple was heard loud and clear by the CSAM community.
    - joe_the_user 5y ago
      I disagree that it's useless. In fact I wouldn't be surprised if it's already been effective despite not being implemented yet. The announcement of an impending threat of scanning probably managed to get most CSAM afficinados to swear off iCloud already. So the idea is that mass surveillance is great even if it's effect is scaring criminals while it actually only catches people at random? We've arrived at the opposite of the common reprise of law. No longer "It is Better that Ten Guilty Persons Go Free Than that One Innocent Person be Convicted" but "It is better that Ten Innocent Persons Be Dragooned Than that One Guilty Person Go Free"
      - simondotau 5y ago
        All I'm saying is that I understand why Apple doesn't want CSAM on their servers, while also reserving the ability to implement end to end encryption on photo galleries in future. It's a tough line for them to walk. Personally I don't see any substantial moral difference to what Apple was planning versus what Google and Facebook already do, aside for the somewhat icky aspect that some initial CPU cycles are occurring on hardware I own. I'm equally concerned that a purported need to scan for CSAM could be used by legislators as a way to convince the population that the Government needs more encryption backdoors. Personally I'd rather have my photos inconsequentially hashed than have encryption legislatively broken.
- chmod775 5y ago
  > Intentional false positives require knowledge of a hash. The code/network detecting this will be run on the client. So yes, a 'hash' or an equivalent will be known. You would have to hope that the secondary system is robust enough to not fall victim to the same manipulated image. And it really is on Apple to first prove it isn't.
  - jdlshore 5y ago
    No, it is not. The hash is computed on the client, but hash comparison happens on the server, using some cryptographic magic that only reveals a “visual derivative” if enough images match. It’s fairly complex and I don’t recall the details, but the library of “bad” hashes is not accessible.
    - tsimionescu 5y ago
      No, the entire premise of NeuralHash is that no information about your pictures leaves your system unless it has been flagged as 'potentially CSAM'. NeuralHash itself is basically a DB of CSAM hashes encoded as an artificial neural network. In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash. Edit: I am almost entirely wrong, see the comment reply from krisoft. Apologies!
      - krisoft 5y ago
        > In your example, each picture on your phone (that you are trying to upload to iCloud) would be hashed, and the hash would be sent to Apple. This is exactly the opposite of what they are promising with NeuralHash. No. You are wrong. They do hash each image and they do upload the hash derived safety voucher always. Here is the Apple technical paper: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... There is a fairly complicated cryptographic algorithm which, according to apple, lets them open the safety voucher only if there is a match with the database. The user devices only have a cryptographicaly blinded version of the hash database which they can’t turn back to real hashes without apple’s private key. This means that the plain neuralhashes are not available to the end users.
- kf6nux 5y ago
  The other reason why false negatives are mostly irrelevant is because the hash DB can be updated to include them, so the risk of storing them in iCloud remains the same.
  - nullc 5y ago
    That doesn't fly. The encrypted list is stored on the devices for security and performance reasons: otherwise it would be too easy to ship people customized lists with arbitrary images. You can't just rapidly revise the list without undermining the privacy protection here. Also the space of modifications is gigantic, for any given image there should be millions of differently hash variations which can be reached without major visual damage. (at least for neuralhash the figure is billions)... so you can't just stick all of them in the database. You could stick widely circulating variations in the database, however... sure.
    - kf6nux 5y ago
      I'm honestly not sure what you're refuting here. Are you saying the hash DB can't be updated or that already uploaded photos won't be compared to hash updates?
      - nullc 5y ago
        The hashdb is part of the operating system and only gets updated with OS updates. It can be updated-- at some cost. But if it is very frequently updated then the protection against receiving a targeted list is lost. To state it another way: Users ask "Why won't the government order apple just gives me a list of the hash of every political image on the internet in order to expose my files?" and the answer is that the database is part of the OS, common to everyone, updated infrequently, so it can't be used in targeted attacks. That protection is lost if the database is frequently updated. It's also true that already uploaded photos won't be rescanned, but that wasn't the point I was trying to make.

nullc 5y ago

The article and kristjansson's post are about intentional false negatives. My comment was pointing out that the well known fact that the neuralhash is highly vulnerable to preimage attacks (intentional false positives) also means that is is highly vulnerable to intentional false negatives. Changing the 96-bit hash to a specific value while minimizing the visible image change is a LOT harder than just changing any single bit, but the harder task has been amply demonstrated. Neuralhash is one component, yes, but it is a limiting component. If a slightly tweaked image has a different NeuralHash it will never be detected by their scheme.

simondotau 5y ago
I don't think Apple is concerned about the <0.1% of CSAM aficionados who read about Apple's plans to scan for CSAM and thinks "hey, let me blindly trust some tool I found on the internet to slightly mangle my highly illegal images so that I can continue co-mingling them with personal photos in iCloud, accessible to anyone who picks up my phone while unlocked. Or any other device synchronised to the same iCloud account." The App Store is presumably full of easy to use apps which can store photo galleries in a separately encrypted vault. This is where people will go after being shooed away from iCloud, not image manglers.

chmod775 5y ago

> being hardened against intentional false positives isn't necessary for it to serve its purpose. Except that every time they get a positive, the device will upload the image to Apple, where at some point an actual human will have to check it. Queue people (4chan) essentially DDOS'ing that infrastructure by turning harmless memes into false positives and distributing them widely. They'll probably automate it too. The chance they won't do such a thing is about 0.01% - if they haven't already. This is just a slight twist on the common prank of making people Google terms related to CSAM. Not only will that cause Apple to violate the privacy of innocent people who were just saving cat pictures from Facebook, it will also require a lot more manpower than Apple is probably anticipating to keep up with the amount of false positives that will come in. And that isn't even considering unintentional false positives. I don't think Apple is planning to start sending out "oopsie we violated your privacy and looked at your memes/family photos" mails to affected people. But it might come to that too once this planned practice is tested against current or future European privacy laws (and such a notification would be the bare minimum in my opinion). That won't go over well once the first people share such an experience.

simondotau 5y ago
This is false. 1. An image is only scanned if it's in your Photo Gallery and about to be uploaded to Apple's servers. 2. The image would need also to match an unspecified second hash algorithm which runs on Apple servers. If it doesn't match this hash as well, it's not considered a match and won't be a candidate for human review. 3. Twenty nine more dual-hash collisions would also need to be imported into their iCloud photo gallery. Human review can only occur if a total of thirty images have matched both hash algorithms. This requirement is cryptographically enforced. 4. Apple human reviewers would only ever see these hash matched images. Which means they'd only ever see some manipulated memes/cat pictures, never any of your personal photos.
- tsimionescu 5y ago
  Your argument works just as well if for any x, NeuralHash(x)=1. The fact is that a 'first line of defense' that is trivial to fool is useless. The fact that there are other lines of defense doesn't mean that the one they would make me run on my own hardware has nay merit whatsoever.

ddevault 5y ago

Say I'm a cop who wants to get into your phone but can't get a warrant. I happen to have an image on my (Android) phone which looks benign to a human, but triggers Apple's CSAM detectors when scanned. I text it to you, the CSAM detector goes off, and suddenly I have probable cause. Cops plant illegal drugs on suspects all the time. Why not this?

pfortuny 5y ago
Imagine a politician receiving dozens of photos which trigger this CSAM algorithm because he has iCloud photos as defaul, and then somehow someone gets wind of the fact… BANG: one less. Try to explain to the public that it is only the first line of defense against child sex and that it does not imply anything at all.
simondotau 5y ago
Because it wouldn't work. 1. You would need at least 30 such images; 2. You'd have to somehow convince your target to import all 30 images into their photo gallery; 3. The chances of the images accidentally passing the human review step is infinitesimal; 4. Unlike a bag of drugs which actually do contain drugs, these fake images would be like police finding a bag of sugar with "FAKE COCAINE FOR MANUFACTURING PROBABLE CAUSE" written on it.