4 ms·

> These cases will be manually reviewed. That is, according to Apple, an Apple employee will then look at your (flagged) pictures. I'm surprised this hasn't go

by avnigo 5y ago

> These cases will be manually reviewed. That is, according to Apple, an Apple employee will then look at your (flagged) pictures. I'm surprised this hasn't gotten enough traction outside of tech news media. Remember the mass celebrity "hacking" of iCloud accounts a few years ago? I wonder how those celebrities would feel knowing that some of their photos may be falsely flagged and shown to other people. And that we expect those humans to act like robots and not sell or leak the photos, etc. Again, I'm surprised we haven't seen a far bigger outcry in the general news media about this yet, but I'm glad to see a lot of articles shining light on how easy it is for false positives and hash collisions to occur, especially at the scale of all iCloud photos.

judge2020 5y ago
They wouldn't be falsely flagged. It doesn't detect naked photos, it detects photos matching real confirmed CSAM based on the NCMEC's database.
- auggierose 5y ago
  If that would always work, a manual review would not be necessary. Just send the flagged photo and its owner straight to the police.
- avnigo 5y ago
  The article posted, as well as many others we've seen recently, demonstrate that collisions are possible, and most likely inevitable with the number of photos to be scanned for iCloud, and Apple recognizes this themselves. It doesn't necessarily mean that all flagged photos would be of explicit content, but even if it's not, is Apple telling us that we should have no expectation of privacy for any photos uploaded to iCloud, after running so many marketing campaigns on privacy? The on-device scanning is also under the guise of privacy too, so they wouldn't have to decrypt the photos on their iCloud servers with the keys they hold (and also save some processing power, maybe).
  - spacedcowboy 5y ago
    Apple already use the same algorithm on photos in email, because email is unencrypted. Last year Apple reported 265 cases according to the NYT. Facebook reported 20.3 million. Devolving the job to the phone is a step to making things more private, not less. Apple don’t need to look at the photos on the server (and all cloud companies in the US are required to inspect photos for CSAM) if it can be done on the phone, removing one more roadblock for why end-to-end encryption hasn’t happened yet.
    - voidnullnil 5y ago
      Am I missing something? Apple says they literally scan stuff locally on your iCrap now and call the police on you if you have $badstuff. Nobody should be having their data scanned in the first place. Is iCloud unencryped? Such a thing exists in 2021? I've been using end to end crypto since 2000. I don't understand why consumers want their devices to do all kinds of special non-utilitarian stuff (I mean I totally understand, it's called politics). This new iCrap is like a toaster that reports you if you put illegally imported bread in it. It will be just like the toaster which will have no measureable impact on illegal imports. Even if $badguys are so dumb to continue using the tech (iCloud???) and lots go to jail, lots more will appear and simply avoid the exact specific cause that sent previous batch to jail. They do not even thave to think. The problem with all this is that everyone is applauding Apple for their bullshit, and so they will applaud the government when they say "oh no, looks like criminals are using non-backdoored data storage methods, what a surprise! we need to make it illegal to have a data storage service without going through a 6 month process to setup a government approved remote auditing service". Then there's also the fact that this is all a pile of experimental crypto [1] being used to solve nothing. Apple has created the exact situation of Cloudflare Pass: they pointlessly made $badip solve a captcha to view a read-only page, and provided a bunch of experimental crypto in a browser plugin to let him use one captcha for multiple domains (they would normally each require their own captcha and corresponding session cookie). They later stopped blocking $badip all together after they realized they are wrong (this took literally 10 years). 1. https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ "CSAM detection" section
    - 5y ago
- josefx 5y ago
  Hashes, no false matches, pick one.
- wongarsu 5y ago
  It will flag pictures that match a perceptual hash of pictures of child abuse. Now what legal kinds of pictures are most similar in composition, color, etc. to those offending pictures? What kinds of pictures would be hardest to distinguish from offending pictures if you were given only 16x16 thumbnails? I'm going to bet the algorithm will struggle the most with exactly the pictures you don't want reviewers or the public to see.
- nullc 5y ago
  If there were no false positives there would be no legitimate reason for Apple to review-- they would just be needlessly exposing their employees to child abuse material. But the fact that there is no legitimate reason according to the system's design doesn't prevent there from being an illegitimate reason: Apple's "review" undermines your legal due process protection against warrantless search. See US v. Ackerman (2016): The appeals court ruled that when AOL forwarded an email with an attachment whos hash matched the NCMEC database to law enforcement without anyone looking at it, and law enforcement looked at the email without obtaining a warrant was an unlawful search and had AOL looked at it first (which they can do by virtue of your agreement with them) and gone "yep, thats child porn" and reported it, it wouldn't have been an unlawful search.
fortran77 5y ago
So we have an Apple employee, the type of person who gets extremely offended over such things as "Chaos Monkeys," deciding if someone is a criminal? No thanks!
lliamander 5y ago
That really alarmed me. I don't think a hosting provider like Apple should have a right to access private pictures, especially just to enforce copyright. Edit: I see now it's not about copyright, but still very disturbing.

by avnigo 5y ago

judge2020 5y ago

They wouldn't be falsely flagged. It doesn't detect naked photos, it detects photos matching real confirmed CSAM based on the NCMEC's database.

auggierose 5y ago
If that would always work, a manual review would not be necessary. Just send the flagged photo and its owner straight to the police.
avnigo 5y ago
The article posted, as well as many others we've seen recently, demonstrate that collisions are possible, and most likely inevitable with the number of photos to be scanned for iCloud, and Apple recognizes this themselves. It doesn't necessarily mean that all flagged photos would be of explicit content, but even if it's not, is Apple telling us that we should have no expectation of privacy for any photos uploaded to iCloud, after running so many marketing campaigns on privacy? The on-device scanning is also under the guise of privacy too, so they wouldn't have to decrypt the photos on their iCloud servers with the keys they hold (and also save some processing power, maybe).
- spacedcowboy 5y ago
  Apple already use the same algorithm on photos in email, because email is unencrypted. Last year Apple reported 265 cases according to the NYT. Facebook reported 20.3 million. Devolving the job to the phone is a step to making things more private, not less. Apple don’t need to look at the photos on the server (and all cloud companies in the US are required to inspect photos for CSAM) if it can be done on the phone, removing one more roadblock for why end-to-end encryption hasn’t happened yet.
  - voidnullnil 5y ago
    Am I missing something? Apple says they literally scan stuff locally on your iCrap now and call the police on you if you have $badstuff. Nobody should be having their data scanned in the first place. Is iCloud unencryped? Such a thing exists in 2021? I've been using end to end crypto since 2000. I don't understand why consumers want their devices to do all kinds of special non-utilitarian stuff (I mean I totally understand, it's called politics). This new iCrap is like a toaster that reports you if you put illegally imported bread in it. It will be just like the toaster which will have no measureable impact on illegal imports. Even if $badguys are so dumb to continue using the tech (iCloud???) and lots go to jail, lots more will appear and simply avoid the exact specific cause that sent previous batch to jail. They do not even thave to think. The problem with all this is that everyone is applauding Apple for their bullshit, and so they will applaud the government when they say "oh no, looks like criminals are using non-backdoored data storage methods, what a surprise! we need to make it illegal to have a data storage service without going through a 6 month process to setup a government approved remote auditing service". Then there's also the fact that this is all a pile of experimental crypto [1] being used to solve nothing. Apple has created the exact situation of Cloudflare Pass: they pointlessly made $badip solve a captcha to view a read-only page, and provided a bunch of experimental crypto in a browser plugin to let him use one captcha for multiple domains (they would normally each require their own captcha and corresponding session cookie). They later stopped blocking $badip all together after they realized they are wrong (this took literally 10 years). 1. https://www.apple.com/child-safety/ https://www.apple.com/child-safety/ "CSAM detection" section
  - 5y ago
josefx 5y ago
Hashes, no false matches, pick one.
wongarsu 5y ago
It will flag pictures that match a perceptual hash of pictures of child abuse. Now what legal kinds of pictures are most similar in composition, color, etc. to those offending pictures? What kinds of pictures would be hardest to distinguish from offending pictures if you were given only 16x16 thumbnails? I'm going to bet the algorithm will struggle the most with exactly the pictures you don't want reviewers or the public to see.
nullc 5y ago
If there were no false positives there would be no legitimate reason for Apple to review-- they would just be needlessly exposing their employees to child abuse material. But the fact that there is no legitimate reason according to the system's design doesn't prevent there from being an illegitimate reason: Apple's "review" undermines your legal due process protection against warrantless search. See US v. Ackerman (2016): The appeals court ruled that when AOL forwarded an email with an attachment whos hash matched the NCMEC database to law enforcement without anyone looking at it, and law enforcement looked at the email without obtaining a warrant was an unlawful search and had AOL looked at it first (which they can do by virtue of your agreement with them) and gone "yep, thats child porn" and reported it, it wouldn't have been an unlawful search.

fortran77 5y ago

So we have an Apple employee, the type of person who gets extremely offended over such things as "Chaos Monkeys," deciding if someone is a criminal? No thanks!

lliamander 5y ago

That really alarmed me. I don't think a hosting provider like Apple should have a right to access private pictures, especially just to enforce copyright. Edit: I see now it's not about copyright, but still very disturbing.