4 ms·

> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Who controls what is in the database? What independent oversight ensures

> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Who controls what is in the database? What independent oversight ensures that it’s only CSAM images? The public certainly can’t audit it. What is stopping the CCP from putting pro-Uighur or Xi Winnie the Pooh images into the database? Or from the US using this to locate images that are interesting from an intelligence perspective (Say for example pictures of Iranian Uranium centrifuges)? Apple says they will only add images that more than one government request? All it would take is a few G men to show up at Apple with a secret court order to do this no? So… China and Hong Kong? The Five Eyes nations?

nullc 5y ago
Their use of a highly vulnerable[1] "neural" perceptual hash function makes the database unauditable: An abusive state actor could obtain child porn images and invisibly alter them to match the hashes of the ideological or ethnically related images they really want to match. If challenged, they could produce child porn images matching their database, and they could had these images to other governments to unknowingly or plausibly denyably include. ...but they don't have to do anything that elaborate because Apple is using powerful cryptography against their users to protect themselves and their data sources from any accountability for the content of the database: The hashes in the database are hidden from everyone who isn't Apple or a state agent. There is no opportunity to learn, much less challenge the content of the database. [1] https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
- azinman2 5y ago
  They have to come from the intersection of two databases from two jurisdictions. So already that’s out as you suggest. Then you’d have to match _nearly exact photos_, which isn’t a vector for general photos of some random minority. Then you’d need 30 of such specific photos, a match with another secret hash, and then a human reviewer at Apple has to say yes it’s CP before anything else happens. I think there are plenty of reasons to be concerned about future laws and future implementations, but let’s be honest about the real risks of this today as it’s currently implemented.
  - nullc 5y ago
    Every step you've described is unfalsifyable: You just have to blindly trust that Apple is doing these things, and that e.g. authoritarian regemes haven't compromised Apple staff with access to the data. > They have to come from the intersection of two databases from two jurisdictions. My message directly answered that. A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database. > Then you’d have to match _nearly exact photos_ It's unclear what you mean here. It's easy to construct completely different images that share a neuralhash. Apple also has no access to the original "child porn" (in quotes because it may not be), as it would be unlawful to provide it to them. > but let’s be honest about the real risks Yes. Lets be honest: Apple has made a decision to reprogram devices owned by their customers to act against their users best interest. They assure us that they will be taking steps to mitigate harm but have used powerful cryptography to conceal their actions and most of their supposed protections are unfalsifable. You're just supposed to explicitly take the word of a party that is already admittedly acting against your best interest. Finally, at best their protections are only moderate. Almost every computer security vulnerability could be dismissed as requiring an impossible series of coincidences, at yet attacks exist.
    - simondotau 5y ago
      > A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database. Even if a state actor constructs an image that is an NeuralHash collision for the material they wish to find, that only gets them through one of the three barriers Apple has erected between your device and the images being reported to a third party. They also need to cause 30 image matches in order to pass threshold secret sharing, and they need to pass human review of these 30 images. Arguably investigation by NCMEC represents a fourth barrier, but I'll ignore that because it's beyond Apple's control. > You just have to blindly trust that Apple This has been true of all closed source operating systems since forever. Functionally, nothing has changed. And whatever you think of the decision Apple has made, you can't argue that they tried to do it in secret.
azinman2 5y ago
National Center for Missing and Exploited Children intersected with a not yet determined db in another jurisdiction, and then human moderators at Apple. So what you’re describing is something that is a concern for the future that could exist anyway (and maybe already does at places like Google that have _zero_ auditable processing of data that already scan for CP from the same database above). But let’s not pretend that’s today.

nullc 5y ago

Their use of a highly vulnerable[1] "neural" perceptual hash function makes the database unauditable: An abusive state actor could obtain child porn images and invisibly alter them to match the hashes of the ideological or ethnically related images they really want to match. If challenged, they could produce child porn images matching their database, and they could had these images to other governments to unknowingly or plausibly denyably include. ...but they don't have to do anything that elaborate because Apple is using powerful cryptography against their users to protect themselves and their data sources from any accountability for the content of the database: The hashes in the database are hidden from everyone who isn't Apple or a state agent. There is no opportunity to learn, much less challenge the content of the database. [1] https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...

azinman2 5y ago
They have to come from the intersection of two databases from two jurisdictions. So already that’s out as you suggest. Then you’d have to match _nearly exact photos_, which isn’t a vector for general photos of some random minority. Then you’d need 30 of such specific photos, a match with another secret hash, and then a human reviewer at Apple has to say yes it’s CP before anything else happens. I think there are plenty of reasons to be concerned about future laws and future implementations, but let’s be honest about the real risks of this today as it’s currently implemented.
- nullc 5y ago
  Every step you've described is unfalsifyable: You just have to blindly trust that Apple is doing these things, and that e.g. authoritarian regemes haven't compromised Apple staff with access to the data. > They have to come from the intersection of two databases from two jurisdictions. My message directly answered that. A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database. > Then you’d have to match _nearly exact photos_ It's unclear what you mean here. It's easy to construct completely different images that share a neuralhash. Apple also has no access to the original "child porn" (in quotes because it may not be), as it would be unlawful to provide it to them. > but let’s be honest about the real risks Yes. Lets be honest: Apple has made a decision to reprogram devices owned by their customers to act against their users best interest. They assure us that they will be taking steps to mitigate harm but have used powerful cryptography to conceal their actions and most of their supposed protections are unfalsifable. You're just supposed to explicitly take the word of a party that is already admittedly acting against your best interest. Finally, at best their protections are only moderate. Almost every computer security vulnerability could be dismissed as requiring an impossible series of coincidences, at yet attacks exist.
  - simondotau 5y ago
    > A state actor can modify an apparent childporn image to match an arbitrarily hash and hand that image to other agencies who will dutifully include it in their database. Even if a state actor constructs an image that is an NeuralHash collision for the material they wish to find, that only gets them through one of the three barriers Apple has erected between your device and the images being reported to a third party. They also need to cause 30 image matches in order to pass threshold secret sharing, and they need to pass human review of these 30 images. Arguably investigation by NCMEC represents a fourth barrier, but I'll ignore that because it's beyond Apple's control. > You just have to blindly trust that Apple This has been true of all closed source operating systems since forever. Functionally, nothing has changed. And whatever you think of the decision Apple has made, you can't argue that they tried to do it in secret.

azinman2 5y ago

National Center for Missing and Exploited Children intersected with a not yet determined db in another jurisdiction, and then human moderators at Apple. So what you’re describing is something that is a concern for the future that could exist anyway (and maybe already does at places like Google that have _zero_ auditable processing of data that already scan for CP from the same database above). But let’s not pretend that’s today.