7 ms·

I really love the EFF, but I also believe the immediate backlash is (relatively) daft. There is a potential for abuse of this system, but consider the following

by Shank 5y ago

I really love the EFF, but I also believe the immediate backlash is (relatively) daft. There is a potential for abuse of this system, but consider the following too: 1. PhotoDNA is already scanning content from Google Photos and a whole host of other service providers. 2. Apple is obviously under pressure to follow suit, but they developed an on-device system, recruited mathematicians to analyze it, and published the results, as well as one in-house proof and one independent proof showing the cryptographic integrity of the system. 3. Nobody, and I mean nobody, is going to successfully convince the general public that a tool designed to stop the spread of CSAM is a "bad thing" unless they can show concrete examples of the abuse. For one and two: given the two options, would you rather that Apple implement serverside scanning, in the clear, or go with the on-device route? If we assume a law was passed to require serverside scanning (which could very well happen), what would that do to privacy? For three: It's an extremely common trope to say that people do things to "save the children." Well, that's still true. Arguing against a CSAM scanning tool, which is technically more privacy preserving than alternatives from other cloud providers, is an extremely uphill battle. The biggest claim here is that the detection tool could be abused against people. And that very well may be possible! But the whole existence of NCMEC is predicated on stopping the active and real danger of child sex exploitation. We know with certainty this is a problem. Compared to a certainty of child sex abuse, the hypothetical risk from such a system is practically laughable to most people. So, I think again, the backlash is daft. It's been about two days of the announcement being public (leaks). The underlying mathematics behind the system has barely been published [0]. It looks like the EFF rushed to make a statement here, and in doing so, it doesn't look like they took the time to analyze the cryptography system, to consider the attacks against it, or to consider possible motivations and outcomes. Maybe they did, and they had advanced access to the material. But it doesn't look like it, and in the court of public opinion, optics are everything. [0]: https://www.apple.com/child-safety/pdf/Alternative_Security_Proof_of_Apple_PSI_System_Mihir_Bellare.pdf https://www.apple.com/child-safety/pdf/Alternative_Security_...

api 5y ago
(2) is important. Apple put effort into making this at least somewhat privacy-respecting, while the other players just scan everything with no limit at all. They also scan everything for any purpose including marketing, political profiling, etc. Apple remains the most privacy respecting major vendor. The only way to do better is fully open software and open hardware.
echelon 5y ago
> There is a potential for abuse of this system, but consider the following too > I think again, the backlash is daft. Don't apologize for this bullshit! Don't let your love of brand trump the reality of what's going on here. Machinery is being put in place to detect what files are on your supposedly secure device. Someone has the reins and promises not to use it for anything other than "protecting the children". How many election cycles or generations does it take to change to an unfavorable climate where this is now a tool of great asymmetrical power to use against the public? What happens when the powers that be see that you downloaded labor union materials, documents from Wikileaks, or other files that implicate you as a risk? Perhaps a content hash on your phone puts you in a flagged bucket where you get pat downs at the airport, increased surveillance, etc. The only position to take here is a full rebuke of Apple. edit: Apple apologists are taking a downright scary position now. I suppose the company has taken a full 180 from their 1984 ad centerpiece. But that's okay, right, because Apple is a part of your identity and it's beyond reproach? edit 2: It's nominally iCloud only (a key feature of the device/ecosystem), but that means having to turn off a lot of settings. One foot in the door... edit 3: Please don't be complicit in allowing this to happen. Don't apologize or rationalize. This is only a first step. We warned that adtech and monitoring and abuse of open source were coming for years, and we were right. We're telling you - loudly - that this will begin a trend of further erosion of privacy and liberty.
- artimaeis 5y ago
  It's not doing any sort of scanning of your photos while they're just sitting on your device. The CSAM scanning only occurs when uploading photos to iCloud, and only to the photos being uploaded. Source (pdf): https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
  - pseudalopex 5y ago
    > It's not doing any sort of scanning of your photos while they're just sitting on your device. Yet. The point is the new system makes it feasible.
vorpalhex 5y ago
Who verifies CSAM databases? Is there a way to verify the CSAM hashlist hasn't been tampered with and additional hashes inserted? Would it be ok to use this approach to stop "terrorism"? Are you ok with both Biden and Trump defining that list?
feanaro 5y ago
> that a tool designed to stop the spread of CSAM is a "bad thing" It's certainly said to be designed to do it, but have you seen concerns raised in the other thread (https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741)? There have been reports from some commenters of the NCMEC database containing unobjectionable photos because they were merely found in a context alongside some CSAM. Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? This is overreach.
- Shank 5y ago
  > Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers. Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now. [0]: https://twitter.com/matthew_d_green/status/1423378285468209153 https://twitter.com/matthew_d_green/status/14233782854682091...
  - shuckles 5y ago
    Matt should read the release before live tweeting FUD. The database is shipped in the iOS image, per the overview, so targeting users is not an issue (roughly).
    - pseudalopex 5y ago
      You should understand the difference between a protocol and an easy to change implementation detail before throwing around words like FUD.
deleted 5y ago
[deleted]
throwaway888abc 5y ago
1. was new to me. TIL - (2014) PhotoDNA Lets Google, FB and Others Hunt Down Child Pornography Without Looking at Your Photos https://petapixel.com/2014/08/08/photodna-lets-google-facebook-others-hunt-down-child-pornography-without-looking-at-your-photos/ https://petapixel.com/2014/08/08/photodna-lets-google-facebo...
wayneftw 5y ago
This is an abuse my property rights. The device is my property and this activity will be using my CPU, battery time and my network bandwidth. That's the abuse right there. They should just use their own computers to do this stuff.
- jdavis703 5y ago
  Then you have two choices, disable iCloud photo backups or don’t upgrade to iOS 15. There are plenty of arguments against Apple’s scheme, but this isn’t one of them.
  - wayneftw 5y ago
    It's the best argument. Every other argument is based on a slippery slope what-if scenario. Also, I pay for iCloud... but they're not paying me for using my phone and bandwidth. I never agreed to that. They can't just pull the rug out from me after we already have an agreement. I mean they can, because I probably "agreed" to it and some fine print garbage EULA, but those fall apart in a court of law.
- samatman 5y ago
  Photos is just an app. You can use another photo app, link it to another cloud provider, and be free of the burden. If you use Photos, you're along for the ride, and you've consented to whatever it does. You don't get a line-item veto on code you choose to run, that's never been how it works. For what it's worth, I'm basically with the EFF on this: it looks like the thin end of a wedge, it sucks and I'm not happy about it. But being histrionic doesn't help anything.
  - jimbob45 5y ago
    I don’t know how true this is. I don’t see any way to block Photos from viewing the files on this device and I see no reason that it can’t read files from my other apps.
  - zionic 5y ago
    No it’s not, it’s the entire OS.
indymike 5y ago
> backlash is daft Fighting to preserve a freedom is not daft, even if it is David vs. Goliath's bigger, meaner brother and his friends.
shivak 5y ago
> recruited mathematicians to analyze it, and published the results, as well as one in-house proof and one independent proof showing the cryptographic integrity of the system. Apple employs cryptographers, but they are not necessarily acting in your interest. Case in point: their use of private set intersection, to preserve privacy..of law enforcement, not users. Their less technical summary: > Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child safety organizations. Apple further transforms this database into an unreadable set of hashes that is securely stored on users’ devices. > Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes. This matching process is powered by a cryptographic technology called private set intersection.. The matching is performed on device, so the user’s privacy isn’t at stake. But, thanks to PSI and the hash preprocessing, the user doesn’t know what law enforcement is looking for.
- xondono 5y ago
  Well, it’d be kind of dumb to make the mistake of building a system to stop child pornography only to have it become the biggest distributor of CP photos in history
  - shivak 5y ago
    Those images are hashed, not transmitted in original format. On top of that, PSI prevents you from learning those hashes, or how many there are. So you can’t tell if the database contains the hash of, say, tank-man.jpg. I understand why this shielding is necessary for the system to work. My point is the crypto is being used to protect law enforcement, not the user.
    - xondono 5y ago
      And my point is that the only way to provide visibility over what is being looked without distributing the material would be to implement some type of ZKP
avnigo 5y ago
I’d be interested to see what any Apple executives would respond to the concerns in interviews, but I don’t expect Apple to issue a press release on the concerns.
cblconfederate 5y ago
What is the point of E2EE vs TLS/SSL based encryption?
randcraw 5y ago
You presume Apple and the DoJ will implement this with human beings at each step. They won't. Both parties will automate as much of this clandestine search as possible. With time, the external visibility and oversight of this practice will fade, and with it, any motivation to confirm fair and accurate matches. Welcome to the sloppiness inherent in clandestine law enforcement intel gathering. As with all politically-motivated initiatives that boldly violate the Constitution (consider the FISA Court, and its rubber stamp approval of 100% of the secret warrants put before it), the use and abuse of this system will go largely underground, like FISA, and its utility will slowly degrade due to lack of oversight. In time, even bad matches will log the IDs of both parties in databases that label them as potential sexual predators. Believe it. That's how modern computer-based gov't intel works. Like most law enforcement policy recommendation systems, Apple's initial match algorithm will never be assessed for accuracy, nor be accountable for being wrong at least 10% of the time. In time it will be replaced by other third party screening software that will be even more poorly written and overseen. That's just what law enforcement does. I've personally seen people suffer this kind of gov't abuse and neglect as a result of clueless automated law enforcement initiatives after 9-1-1. I don't welcome more, nor the gradual and willful tossing of everyone's basic Constitutional rights that Apple's practice portends. The damages to personal liberty that are inherent in conducting secret searches without cause or oversight is exactly why the Fourth Amendment requires a warrant before conducting a search. NOW is the time to disabuse your sense of 'daftness'; not years from now, after the Fourth and Fifth Amendments become irreversibly passe. Or should I say, 'daft'?

by Shank 5y ago

api 5y ago

(2) is important. Apple put effort into making this at least somewhat privacy-respecting, while the other players just scan everything with no limit at all. They also scan everything for any purpose including marketing, political profiling, etc. Apple remains the most privacy respecting major vendor. The only way to do better is fully open software and open hardware.

echelon 5y ago

> There is a potential for abuse of this system, but consider the following too > I think again, the backlash is daft. Don't apologize for this bullshit! Don't let your love of brand trump the reality of what's going on here. Machinery is being put in place to detect what files are on your supposedly secure device. Someone has the reins and promises not to use it for anything other than "protecting the children". How many election cycles or generations does it take to change to an unfavorable climate where this is now a tool of great asymmetrical power to use against the public? What happens when the powers that be see that you downloaded labor union materials, documents from Wikileaks, or other files that implicate you as a risk? Perhaps a content hash on your phone puts you in a flagged bucket where you get pat downs at the airport, increased surveillance, etc. The only position to take here is a full rebuke of Apple. edit: Apple apologists are taking a downright scary position now. I suppose the company has taken a full 180 from their 1984 ad centerpiece. But that's okay, right, because Apple is a part of your identity and it's beyond reproach? edit 2: It's nominally iCloud only (a key feature of the device/ecosystem), but that means having to turn off a lot of settings. One foot in the door... edit 3: Please don't be complicit in allowing this to happen. Don't apologize or rationalize. This is only a first step. We warned that adtech and monitoring and abuse of open source were coming for years, and we were right. We're telling you - loudly - that this will begin a trend of further erosion of privacy and liberty.

artimaeis 5y ago
It's not doing any sort of scanning of your photos while they're just sitting on your device. The CSAM scanning only occurs when uploading photos to iCloud, and only to the photos being uploaded. Source (pdf): https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- pseudalopex 5y ago
  > It's not doing any sort of scanning of your photos while they're just sitting on your device. Yet. The point is the new system makes it feasible.

vorpalhex 5y ago

Who verifies CSAM databases? Is there a way to verify the CSAM hashlist hasn't been tampered with and additional hashes inserted? Would it be ok to use this approach to stop "terrorism"? Are you ok with both Biden and Trump defining that list?

feanaro 5y ago

> that a tool designed to stop the spread of CSAM is a "bad thing" It's certainly said to be designed to do it, but have you seen concerns raised in the other thread (https://news.ycombinator.com/item?id=28068741 https://news.ycombinator.com/item?id=28068741)? There have been reports from some commenters of the NCMEC database containing unobjectionable photos because they were merely found in a context alongside some CSAM. Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? This is overreach.

Shank 5y ago
> Who audits these databases? Where is the oversight to guarantee only appropriate content is included? They are famously opaque because the very viewing of the content is illegal. So how can we know that they contain what they are purported to contain? I wholeheartedly agree: there is an audit question here too. The contents of the database are by far the most dangerous part of this equation, malicious or not, targeted or not. I don't like the privacy implications about this, nor the potential for abuse. I would love to see some kind of way to audit the database, or ensure that it's only used "for good." I just don't know what that system is, and I know that PhotoDNA is already in use on other cloud providers. Matthew Green's ongoing analysis [0] is really worth keeping an eye on. For example, there's a good question: can you just scan against a different database for different people? These are the right questions given what we have right now. [0]: https://twitter.com/matthew_d_green/status/1423378285468209153 https://twitter.com/matthew_d_green/status/14233782854682091...
- shuckles 5y ago
  Matt should read the release before live tweeting FUD. The database is shipped in the iOS image, per the overview, so targeting users is not an issue (roughly).
  - pseudalopex 5y ago
    You should understand the difference between a protocol and an easy to change implementation detail before throwing around words like FUD.

deleted 5y ago

[deleted]

throwaway888abc 5y ago

1. was new to me. TIL - (2014) PhotoDNA Lets Google, FB and Others Hunt Down Child Pornography Without Looking at Your Photos https://petapixel.com/2014/08/08/photodna-lets-google-facebook-others-hunt-down-child-pornography-without-looking-at-your-photos/ https://petapixel.com/2014/08/08/photodna-lets-google-facebo...

wayneftw 5y ago

This is an abuse my property rights. The device is my property and this activity will be using my CPU, battery time and my network bandwidth. That's the abuse right there. They should just use their own computers to do this stuff.

jdavis703 5y ago
Then you have two choices, disable iCloud photo backups or don’t upgrade to iOS 15. There are plenty of arguments against Apple’s scheme, but this isn’t one of them.
- wayneftw 5y ago
  It's the best argument. Every other argument is based on a slippery slope what-if scenario. Also, I pay for iCloud... but they're not paying me for using my phone and bandwidth. I never agreed to that. They can't just pull the rug out from me after we already have an agreement. I mean they can, because I probably "agreed" to it and some fine print garbage EULA, but those fall apart in a court of law.
samatman 5y ago
Photos is just an app. You can use another photo app, link it to another cloud provider, and be free of the burden. If you use Photos, you're along for the ride, and you've consented to whatever it does. You don't get a line-item veto on code you choose to run, that's never been how it works. For what it's worth, I'm basically with the EFF on this: it looks like the thin end of a wedge, it sucks and I'm not happy about it. But being histrionic doesn't help anything.
- jimbob45 5y ago
  I don’t know how true this is. I don’t see any way to block Photos from viewing the files on this device and I see no reason that it can’t read files from my other apps.
- zionic 5y ago
  No it’s not, it’s the entire OS.

indymike 5y ago

> backlash is daft Fighting to preserve a freedom is not daft, even if it is David vs. Goliath's bigger, meaner brother and his friends.

shivak 5y ago

> recruited mathematicians to analyze it, and published the results, as well as one in-house proof and one independent proof showing the cryptographic integrity of the system. Apple employs cryptographers, but they are not necessarily acting in your interest. Case in point: their use of private set intersection, to preserve privacy..of law enforcement, not users. Their less technical summary: > Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child safety organizations. Apple further transforms this database into an unreadable set of hashes that is securely stored on users’ devices. > Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes. This matching process is powered by a cryptographic technology called private set intersection.. The matching is performed on device, so the user’s privacy isn’t at stake. But, thanks to PSI and the hash preprocessing, the user doesn’t know what law enforcement is looking for.

xondono 5y ago
Well, it’d be kind of dumb to make the mistake of building a system to stop child pornography only to have it become the biggest distributor of CP photos in history
- shivak 5y ago
  Those images are hashed, not transmitted in original format. On top of that, PSI prevents you from learning those hashes, or how many there are. So you can’t tell if the database contains the hash of, say, tank-man.jpg. I understand why this shielding is necessary for the system to work. My point is the crypto is being used to protect law enforcement, not the user.
  - xondono 5y ago
    And my point is that the only way to provide visibility over what is being looked without distributing the material would be to implement some type of ZKP

avnigo 5y ago

I’d be interested to see what any Apple executives would respond to the concerns in interviews, but I don’t expect Apple to issue a press release on the concerns.

cblconfederate 5y ago

What is the point of E2EE vs TLS/SSL based encryption?

randcraw 5y ago

You presume Apple and the DoJ will implement this with human beings at each step. They won't. Both parties will automate as much of this clandestine search as possible. With time, the external visibility and oversight of this practice will fade, and with it, any motivation to confirm fair and accurate matches. Welcome to the sloppiness inherent in clandestine law enforcement intel gathering. As with all politically-motivated initiatives that boldly violate the Constitution (consider the FISA Court, and its rubber stamp approval of 100% of the secret warrants put before it), the use and abuse of this system will go largely underground, like FISA, and its utility will slowly degrade due to lack of oversight. In time, even bad matches will log the IDs of both parties in databases that label them as potential sexual predators. Believe it. That's how modern computer-based gov't intel works. Like most law enforcement policy recommendation systems, Apple's initial match algorithm will never be assessed for accuracy, nor be accountable for being wrong at least 10% of the time. In time it will be replaced by other third party screening software that will be even more poorly written and overseen. That's just what law enforcement does. I've personally seen people suffer this kind of gov't abuse and neglect as a result of clueless automated law enforcement initiatives after 9-1-1. I don't welcome more, nor the gradual and willful tossing of everyone's basic Constitutional rights that Apple's practice portends. The damages to personal liberty that are inherent in conducting secret searches without cause or oversight is exactly why the Fourth Amendment requires a warrant before conducting a search. NOW is the time to disabuse your sense of 'daftness'; not years from now, after the Fourth and Fifth Amendments become irreversibly passe. Or should I say, 'daft'?