6 ms·

Does it? This just gets things flagged for another test and then manual review, it doesn't flag you as a pedophile or something. It seems to be getting a lot

by Natsu 5y ago

Does it? This just gets things flagged for another test and then manual review, it doesn't flag you as a pedophile or something. It seems to be getting a lot of attention because it's an easy target, but it doesn't seem to be a very meaningful one given the system they say they use.

setr 5y ago
Any notion of privacy is eliminated, if you can reasonably expect arbitrary photos to be selected and sent to apple for manual review (where selection could make up really any % of the photos you upload, and presumably you won’t be told which ones were flagged for review). It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)
- selsta 5y ago
  This has always been the case when uploading something to the cloud unencrypted. You never had any privacy in the first place.
  - vineyardmike 5y ago
    But this isn't uploaded to the cloud. Its your device.
    - axoltl 5y ago
      It’s only done to pictures that you’re about to upload to the cloud.
      - torstenvl 5y ago
        A claim that can't be independently verified by the owner of the device. Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.
        nicce 5y ago
        With that argument, how do you know that Apple hasn't been doing this already for years? You either fully trust Apple and use their devices, or not.
        vineyardmike 5y ago
        Now we know that the technology is built, so while we don't know what they use this for, they've told us they have it and its used for something. Its not hard to assume they'd be less vocal about future changes.
      - vineyardmike 5y ago
        The point is that its done on our end, not apple's end. Many reasons this is bad. Obviously the "Slippery slope" argument around it scanning more in future and being abused by the state. There is the possibility that its exploited/bug that sends data when it shouldn't or falsely reports you (not a collision). Eg. a bug that reports all photos as bad instead of listening to results of scan. Obviously the possibility of collision sucks. But thats somewhat constant across implementation options. It wastes your CPU resources and network resources to do something that only benefits apple. It's a bad precedent because other companies will likely do this now, repeated the above arguments with increasingly less trustworthy companies.
        nicce 5y ago
        Bugs are kinda slippery slope arguments as well, since they are impossible to prevent or measure with 100% accuracy. These are stopped in human review. (Hopefully) It seems that this system might be extremly resistant for collision to prevent them ending up to human review. https://news.ycombinator.com/item?id=28305946 https://news.ycombinator.com/item?id=28305946 > It wastes your CPU resources and network resources to do something that only benefits apple Okay, how practical is this problem? iCloud already scans your files to know whether they are need to be synced. And this scan applies lesser times, only once before upload per file. Encryption does not mess with the filesize, so we are talking about few bytes of extra metadata per image here. Regular image with raw format is tens of megabytes.
        vineyardmike 5y ago
        > Okay, how practical is this problem? Mobile phones operate off battery and can be very important tools for safety and navigation. Every wasted minute of battery is important. > Encryption does not mess with the filesize No but giant bloom filters take storage.
        snet0 5y ago
        Only benefits Apple?
echelon 5y ago
It could flag you as a supporter of Free Hong Kong in China, women's rights in Afghanistan, or maybe just someone who likes gay porn in Russia. Building these types of systems is antithetical to a free society. Can you predict the US' political climate in twenty years' time? No? Then don't build this.
- djrogers 5y ago
  > It could flag you as a supporter of Free Hong Kong in China, women's rights in Afghanistan, or maybe just someone who likes gay porn in Russia And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? The hash DB will only include photos that are in NCMEC and a second countries’ CSAM database. And it will be trivial to verify that the hash DB is consistent across different countries.
  - matheusmoreira 5y ago
    > And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? All they have to do is say: in order to sell iPhones in this country, you must do what we want. Apple will capitulate.
    - dkonofalski 5y ago
      I know it means nothing to people that make this argument but Apple have already said that they won't give in to those demands. If they do, I would expect far more backlash than from just one country.
      - matheusmoreira 5y ago
        Yeah, Apple will totally refuse the chinese government which owns the chinese market that Apple cannot afford to lose.
        dkonofalski 5y ago
prophesi 5y ago
It's easy to imagine innocuous photos with nude coloring being altered to match CSAM hashes. Having a few will flag your account for human review. The low res visual derivative[0][1] that the human moderators see will look credible enough to be CP[2] and alert the authorities. [0] It's federally illegal for anyone to have CP on their system, or to view it. [1] So the workaround is displaying a distorted low-res version instead https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model... [2] You can replace CP with anything your oppressive regime is trying to snuff out.
- dkonofalski 5y ago
  >Having a few will flag your account for human review. This is not correct. Having a few will put those photos through a second, blind hashing algorithm. It wouldn't be flagged for human review unless the images in question passed the second stage and, even then, you wouldn't have any of the information for those hashes to match.
  - prophesi 5y ago
    My bad. Having a few will have the flagged photos sent to their servers, now liable to be obtained by external forces. Then processed by a second visual hashing algorithm, liable to the same type of collision attacks, using obfuscation as its promise of security.
    - dkonofalski 5y ago
      It doesn't send the photos to their servers. Read the white paper before you make snarky responses. Most of what you're saying is covered there.
      - prophesi 5y ago
        They send a low res visual derivative. Which is worse than sending the actual photo in regards to false positives.
gowld 5y ago
"Manual Review" of what? My private photos, including all my legal but unpopular sexual and political activity? Including my wrongly-illegal private herbal activity?
- conception 5y ago
  The photo specifically flagged by a hash that you purposefully saved that was going to sync to your iCloud photo library.
- mono-bob 5y ago
  Not even that. Apple recieves a visual derivative of the image, such as a low-resolution version, for manual review. The downside it that this makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with. Images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.
mulmen 5y ago
The matches get sent to LEO for that validation. LEO has no incentive to exonerate you, their incentives are purely to identify wrongdoing. Any wrongdoing. Having a image sent to LEO is effectively being accused of a crime. What prevents a judge from approving a search warrant on the basis of an image that matched a hash? The answer is, of course, “nothing”. What guarantee do we have that these matches aren’t used as a signal to some other automated extra-judicial search? Again, none. None of this is fantasy. Take a look at the no-fly list and and no-knock warrants and ask yourself if you really want an automated system making accusations to LEO. e: clarified that images are sent to LEO, not just hashes.
- varispeed 5y ago
  "There is no smoke without a fire" "Those targets will not meet themselves" "What if that person has CSAM but tried to obfuscate it. Would you forgive yourself if you let it go and it turned out you were wrong?"
- ec109685 5y ago
  No, it doesn't get sent to LEO. It gets run against another hash function that can't be guessed using this method, and an apple employee looks over it.
  - mulmen 5y ago
    Thanks, I updated my post to clarify that images are sent to LEO, not hashes.
- Natsu 5y ago
  I can see it being used to forcibly start an investigation of someone, but the image itself is not CP and would be quickly proven as such assuming you have a halfway competent lawyer. There's a fair point in there about public defenders not meeting that bar, at least in terms of the quality of their representation given the amount of work they have, though.
  - mulmen 5y ago
matheusmoreira 5y ago
False positives can make your life a living hell by bringing the authorities down on you and then requiring you to prove your innocence to them. People are routinely arrested over the results of field drug tests despite their high sensitivity and high probability of false positives.
- xadhominemx 5y ago
  Why would it bring the authorities onto you? It flags the photos for manual review.
  - mono-bob 5y ago
    Apple recieves a visual derivative of the image, such as a low-resolution version, for manual review. This makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with, images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.
    - xadhominemx 5y ago
      So your concern is that someone is unlucky enough to a) suffer 30+ neural hash collisions, b) some subset of those are indistinguishable from child pornography due to low resolution (flesh colored bathing suits), c) those images match the multi stage hash, then d) authorities subpoena the full resolution images, which turn out to be innocuous but they decide to harass the user anyway. I dunno man, I’m not to worried about this chain of events.
boudin 5y ago
There's so much wrong with all of this. First automation of surveillance. I know Apple is not the first, but having private companies slowly treating all its customers as suspect is really wrong. We're not talking of someone suspected, with targeted surveillance. We're talking about everybody, USSR style. I don't care that it's automated. We can go through the safeguards that Apple put in place. Second hash? Seems that's just make it more costly to generate a collision, it's absolutly not a guarantee. Human verification step? Just look at the shitshow of apple store review and how incompetent Apple is at dealing with this stage, that makes it even worse and more arbitrary. There's nothing to defend at all in this whole thing. What Apple is putting in place is dangerous. Using pedopornography as an excuse to set a global surveillance system is horrible.

by Natsu 5y ago

setr 5y ago

Any notion of privacy is eliminated, if you can reasonably expect arbitrary photos to be selected and sent to apple for manual review (where selection could make up really any % of the photos you upload, and presumably you won’t be told which ones were flagged for review). It’s getting a lot of attention because this trivially violates the last 5 years of apple marketing/branding, and as per usual, by depending on “won’t someone think of the children!” logic (which basically justifies anything and everything)

selsta 5y ago
This has always been the case when uploading something to the cloud unencrypted. You never had any privacy in the first place.
- vineyardmike 5y ago
  But this isn't uploaded to the cloud. Its your device.
  - axoltl 5y ago
    It’s only done to pictures that you’re about to upload to the cloud.
    - torstenvl 5y ago
      A claim that can't be independently verified by the owner of the device. Apple isn't exactly known for software transparency or allowing software that monitors iOS internals.
      - nicce 5y ago
        With that argument, how do you know that Apple hasn't been doing this already for years? You either fully trust Apple and use their devices, or not.
        vineyardmike 5y ago
        Now we know that the technology is built, so while we don't know what they use this for, they've told us they have it and its used for something. Its not hard to assume they'd be less vocal about future changes.
    - vineyardmike 5y ago
      The point is that its done on our end, not apple's end. Many reasons this is bad. Obviously the "Slippery slope" argument around it scanning more in future and being abused by the state. There is the possibility that its exploited/bug that sends data when it shouldn't or falsely reports you (not a collision). Eg. a bug that reports all photos as bad instead of listening to results of scan. Obviously the possibility of collision sucks. But thats somewhat constant across implementation options. It wastes your CPU resources and network resources to do something that only benefits apple. It's a bad precedent because other companies will likely do this now, repeated the above arguments with increasingly less trustworthy companies.
      - nicce 5y ago
        Bugs are kinda slippery slope arguments as well, since they are impossible to prevent or measure with 100% accuracy. These are stopped in human review. (Hopefully) It seems that this system might be extremly resistant for collision to prevent them ending up to human review. https://news.ycombinator.com/item?id=28305946 https://news.ycombinator.com/item?id=28305946 > It wastes your CPU resources and network resources to do something that only benefits apple Okay, how practical is this problem? iCloud already scans your files to know whether they are need to be synced. And this scan applies lesser times, only once before upload per file. Encryption does not mess with the filesize, so we are talking about few bytes of extra metadata per image here. Regular image with raw format is tens of megabytes.
        vineyardmike 5y ago
        > Okay, how practical is this problem? Mobile phones operate off battery and can be very important tools for safety and navigation. Every wasted minute of battery is important. > Encryption does not mess with the filesize No but giant bloom filters take storage.
      - snet0 5y ago
        Only benefits Apple?

echelon 5y ago

It could flag you as a supporter of Free Hong Kong in China, women's rights in Afghanistan, or maybe just someone who likes gay porn in Russia. Building these types of systems is antithetical to a free society. Can you predict the US' political climate in twenty years' time? No? Then don't build this.

djrogers 5y ago
> It could flag you as a supporter of Free Hong Kong in China, women's rights in Afghanistan, or maybe just someone who likes gay porn in Russia And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? The hash DB will only include photos that are in NCMEC and a second countries’ CSAM database. And it will be trivial to verify that the hash DB is consistent across different countries.
- matheusmoreira 5y ago
  > And how do you propose that all of these evil regimes are going to get their images into the NCMEC database? All they have to do is say: in order to sell iPhones in this country, you must do what we want. Apple will capitulate.
  - dkonofalski 5y ago
    I know it means nothing to people that make this argument but Apple have already said that they won't give in to those demands. If they do, I would expect far more backlash than from just one country.
    - matheusmoreira 5y ago
      Yeah, Apple will totally refuse the chinese government which owns the chinese market that Apple cannot afford to lose.
      - dkonofalski 5y ago

prophesi 5y ago

It's easy to imagine innocuous photos with nude coloring being altered to match CSAM hashes. Having a few will flag your account for human review. The low res visual derivative[0][1] that the human moderators see will look credible enough to be CP[2] and alert the authorities. [0] It's federally illegal for anyone to have CP on their system, or to view it. [1] So the workaround is displaying a distorted low-res version instead https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf https://www.apple.com/child-safety/pdf/Security_Threat_Model... [2] You can replace CP with anything your oppressive regime is trying to snuff out.

dkonofalski 5y ago
>Having a few will flag your account for human review. This is not correct. Having a few will put those photos through a second, blind hashing algorithm. It wouldn't be flagged for human review unless the images in question passed the second stage and, even then, you wouldn't have any of the information for those hashes to match.
- prophesi 5y ago
  My bad. Having a few will have the flagged photos sent to their servers, now liable to be obtained by external forces. Then processed by a second visual hashing algorithm, liable to the same type of collision attacks, using obfuscation as its promise of security.
  - dkonofalski 5y ago
    It doesn't send the photos to their servers. Read the white paper before you make snarky responses. Most of what you're saying is covered there.
    - prophesi 5y ago
      They send a low res visual derivative. Which is worse than sending the actual photo in regards to false positives.

gowld 5y ago

"Manual Review" of what? My private photos, including all my legal but unpopular sexual and political activity? Including my wrongly-illegal private herbal activity?

conception 5y ago
The photo specifically flagged by a hash that you purposefully saved that was going to sync to your iCloud photo library.
mono-bob 5y ago
Not even that. Apple recieves a visual derivative of the image, such as a low-resolution version, for manual review. The downside it that this makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with. Images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.

mulmen 5y ago

The matches get sent to LEO for that validation. LEO has no incentive to exonerate you, their incentives are purely to identify wrongdoing. Any wrongdoing. Having a image sent to LEO is effectively being accused of a crime. What prevents a judge from approving a search warrant on the basis of an image that matched a hash? The answer is, of course, “nothing”. What guarantee do we have that these matches aren’t used as a signal to some other automated extra-judicial search? Again, none. None of this is fantasy. Take a look at the no-fly list and and no-knock warrants and ask yourself if you really want an automated system making accusations to LEO. e: clarified that images are sent to LEO, not just hashes.

varispeed 5y ago
"There is no smoke without a fire" "Those targets will not meet themselves" "What if that person has CSAM but tried to obfuscate it. Would you forgive yourself if you let it go and it turned out you were wrong?"
ec109685 5y ago
No, it doesn't get sent to LEO. It gets run against another hash function that can't be guessed using this method, and an apple employee looks over it.
- mulmen 5y ago
  Thanks, I updated my post to clarify that images are sent to LEO, not hashes.
Natsu 5y ago
I can see it being used to forcibly start an investigation of someone, but the image itself is not CP and would be quickly proven as such assuming you have a halfway competent lawyer. There's a fair point in there about public defenders not meeting that bar, at least in terms of the quality of their representation given the amount of work they have, though.
- mulmen 5y ago

matheusmoreira 5y ago

False positives can make your life a living hell by bringing the authorities down on you and then requiring you to prove your innocence to them. People are routinely arrested over the results of field drug tests despite their high sensitivity and high probability of false positives.

xadhominemx 5y ago
Why would it bring the authorities onto you? It flags the photos for manual review.
- mono-bob 5y ago
  Apple recieves a visual derivative of the image, such as a low-resolution version, for manual review. This makes the human review process tricky. Because there is, understandably, no actual CSAM image to compare the visual derivative with, images that look similar enough (contain nudity or a child in a swimsuit) may get past the human review process.
  - xadhominemx 5y ago
    So your concern is that someone is unlucky enough to a) suffer 30+ neural hash collisions, b) some subset of those are indistinguishable from child pornography due to low resolution (flesh colored bathing suits), c) those images match the multi stage hash, then d) authorities subpoena the full resolution images, which turn out to be innocuous but they decide to harass the user anyway. I dunno man, I’m not to worried about this chain of events.

boudin 5y ago

There's so much wrong with all of this. First automation of surveillance. I know Apple is not the first, but having private companies slowly treating all its customers as suspect is really wrong. We're not talking of someone suspected, with targeted surveillance. We're talking about everybody, USSR style. I don't care that it's automated. We can go through the safeguards that Apple put in place. Second hash? Seems that's just make it more costly to generate a collision, it's absolutly not a guarantee. Human verification step? Just look at the shitshow of apple store review and how incompetent Apple is at dealing with this stage, that makes it even worse and more arbitrary. There's nothing to defend at all in this whole thing. What Apple is putting in place is dangerous. Using pedopornography as an excuse to set a global surveillance system is horrible.