4 ms·

Right? Should cars have facial recognition cameras so they don't enable child trafficking?

nicce 5y ago
Not relevant example as Apple is only required to stop redistribution of existing material. They are not forced to look for new material.
- cvwright 5y ago
  But the point is, once you accept something noble and difficult like "preventing CSAM" as your primary overriding goal, then there's nothing that's too far or too extreme if it will help you with your noble goal. Five years ago, the idea of Apple scanning photos on your phone would have been absurd. Five years from now, what will people think about hotels installing AI-powered cameras in every room? The vendor swears they only start recording when they detect an act of abuse. It sounds absurd now, but where do you draw the line?
  - nicce 5y ago
    Many (maybe 5) years ago Apple launched Neural Net to categorize your photos and this scans all of them, a lot whether they are in the cloud or not. Difference is, that we don’t know where this information is stored. Still nobody is worried about that. Feature, which allows more than this newly added CSAM functionality. If someone wants to misuse that in hidden, there is no difference of now or future. Because all we have is trust. Speculation suddenly raises, when common politic reasons are mentioned. It does not really matter if the scanning happens on device or iCloud in this situation, because you have to always trust their closed source system. Google has scanned your images since 2009 in the cloud unencrypted, but now when Apple makes situation better, it is suddenly bad. All tools have been out there already. There are no really other options to get more privacy than this, but people refuse to see that. Well, there is voting. Vote people who puts privacy over everything. That would make everything easy.
    - giantrobot 5y ago
      > Google has scanned your images since 2009 in the cloud unencrypted, but now when Apple makes situation better, it is suddenly bad. At the moment Apple's scanning policy is about the same as it was before. They claim they're only scanning photos if iCloud photos are enabled. The change they're advertising is doing the actual scanning process locally. The problem is two fold. The first is Apple went from scanning only explicitly uploaded content to local content. Since they've decided to intrude on local content once "for the children" it's not out of the realm of possibility (if not likely) they will make further intrusions in the future for prima facie noble reasons. Are third party apps going to be restricted on saving data unless they allow access to Apple's CSAM scanner? Will it start scanning texts or e-mails tomorrow letting and rando flood a person's phone with CSAM and get them arrested? Adding a local scanning system like this is a slippery slope. The second problem is the opaqueness of the system. This has multiple sub-problems. While the NCMEC has a laudable goal, involving them in the CSAM scanning process involves an outsize level of trust I don't think they have earned. They have law enforcement's unfortunate disdain for personal privacy coupled with a fanatical devotion to their cause. They believe their actions are always correct and just so long as they supposedly serve their goal of "protecting children". Due to the opaque nature of their content library it's not crazy to think repressive regimes will get self-serving content added to the source libraries for CSAM scanning. There's plenty of places where homosexuality is punishable by death and even mildly anti-government content will land you in jail. Obviously you and I can't go look at NCMEC/ICMEC CSAM libraries to check for falsely added content. So how are we supposed to trust a system run by fanatics to not have simple errors? Which leads to the other opaqueness sub-problem. Apple's design is interesting, if not laudable, but is closed source and full of black boxes. PhotoDNA, NeuralHash, and the like are not published algorithms anyone can verify. We don't even have a way of knowing if some image we have tripped a false positive and have to trust Apple's unknown "threshold" isn't 1. So not only does the public, the subject of these new intrusions, have no way of auditing the database but they have no way of auditing the code or process. A stupid bug in the scanning system could get a user reported to Apple which we then have to trust not to forward (and not to have additional bugs in their reporting system) them to law enforcement and ruin their life. So I am concerned with scope creep and bugs/false positives. I can live with a bug that causes video playback to stutter or a black box system in Maps that gives me the wrong hours for a restaurant. It's much harder to live with bugs that can get me arrested or even killed thanks to trigger happy police. Apple's system might be technically adept but their promises of future behavior aren't trustworthy since they've already changed their behavior with this new system.
- Zak 5y ago
  Apple isn't required to stop redistribution of existing material; they're required to make a report if they have actual knowledge of users possessing or distributing apparent CSAM. This is different from what your comment implies in two ways. First, they do not have an obligation to actively look for CSAM; they only incur an obligation if they find it. Second, the obligation applies to apparent illegal content rather than known illegal content. What qualifies as apparent could end up in court. https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title18-section2258A&num=0&edition=prelim https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim...
  - nicce 5y ago
    > Apple isn't required to stop redistribution of existing material; they're required to make a report if they have actual knowledge of users possessing or distributing apparent CSAM. This isn't that simple. If NCMEC comes with the properties of CSAM (e.g. hashes) and asks provider especially those to be removed from their cloud, it is hard to remove them without looking for them. This is different than an obligation to actively look for CSAM in general.
    - Zak 5y ago
      Can you cite a statute that requires a provider to look for hashes when NCMEC asks them to? If NCMEC told a provider that a specific URL (or similarly unique identifier) contains CSAM, the provider would be obligated to destroy the associated file or be guilty of possession/distribution because at that point they know what they have. That's different from NCMEC providing hashes that could identify files the provider may or may not be storing.

Not relevant example as Apple is only required to stop redistribution of existing material. They are not forced to look for new material.

cvwright 5y ago
But the point is, once you accept something noble and difficult like "preventing CSAM" as your primary overriding goal, then there's nothing that's too far or too extreme if it will help you with your noble goal. Five years ago, the idea of Apple scanning photos on your phone would have been absurd. Five years from now, what will people think about hotels installing AI-powered cameras in every room? The vendor swears they only start recording when they detect an act of abuse. It sounds absurd now, but where do you draw the line?
- nicce 5y ago
  Many (maybe 5) years ago Apple launched Neural Net to categorize your photos and this scans all of them, a lot whether they are in the cloud or not. Difference is, that we don’t know where this information is stored. Still nobody is worried about that. Feature, which allows more than this newly added CSAM functionality. If someone wants to misuse that in hidden, there is no difference of now or future. Because all we have is trust. Speculation suddenly raises, when common politic reasons are mentioned. It does not really matter if the scanning happens on device or iCloud in this situation, because you have to always trust their closed source system. Google has scanned your images since 2009 in the cloud unencrypted, but now when Apple makes situation better, it is suddenly bad. All tools have been out there already. There are no really other options to get more privacy than this, but people refuse to see that. Well, there is voting. Vote people who puts privacy over everything. That would make everything easy.
  - giantrobot 5y ago
    > Google has scanned your images since 2009 in the cloud unencrypted, but now when Apple makes situation better, it is suddenly bad. At the moment Apple's scanning policy is about the same as it was before. They claim they're only scanning photos if iCloud photos are enabled. The change they're advertising is doing the actual scanning process locally. The problem is two fold. The first is Apple went from scanning only explicitly uploaded content to local content. Since they've decided to intrude on local content once "for the children" it's not out of the realm of possibility (if not likely) they will make further intrusions in the future for prima facie noble reasons. Are third party apps going to be restricted on saving data unless they allow access to Apple's CSAM scanner? Will it start scanning texts or e-mails tomorrow letting and rando flood a person's phone with CSAM and get them arrested? Adding a local scanning system like this is a slippery slope. The second problem is the opaqueness of the system. This has multiple sub-problems. While the NCMEC has a laudable goal, involving them in the CSAM scanning process involves an outsize level of trust I don't think they have earned. They have law enforcement's unfortunate disdain for personal privacy coupled with a fanatical devotion to their cause. They believe their actions are always correct and just so long as they supposedly serve their goal of "protecting children". Due to the opaque nature of their content library it's not crazy to think repressive regimes will get self-serving content added to the source libraries for CSAM scanning. There's plenty of places where homosexuality is punishable by death and even mildly anti-government content will land you in jail. Obviously you and I can't go look at NCMEC/ICMEC CSAM libraries to check for falsely added content. So how are we supposed to trust a system run by fanatics to not have simple errors? Which leads to the other opaqueness sub-problem. Apple's design is interesting, if not laudable, but is closed source and full of black boxes. PhotoDNA, NeuralHash, and the like are not published algorithms anyone can verify. We don't even have a way of knowing if some image we have tripped a false positive and have to trust Apple's unknown "threshold" isn't 1. So not only does the public, the subject of these new intrusions, have no way of auditing the database but they have no way of auditing the code or process. A stupid bug in the scanning system could get a user reported to Apple which we then have to trust not to forward (and not to have additional bugs in their reporting system) them to law enforcement and ruin their life. So I am concerned with scope creep and bugs/false positives. I can live with a bug that causes video playback to stutter or a black box system in Maps that gives me the wrong hours for a restaurant. It's much harder to live with bugs that can get me arrested or even killed thanks to trigger happy police. Apple's system might be technically adept but their promises of future behavior aren't trustworthy since they've already changed their behavior with this new system.
Zak 5y ago
Apple isn't required to stop redistribution of existing material; they're required to make a report if they have actual knowledge of users possessing or distributing apparent CSAM. This is different from what your comment implies in two ways. First, they do not have an obligation to actively look for CSAM; they only incur an obligation if they find it. Second, the obligation applies to apparent illegal content rather than known illegal content. What qualifies as apparent could end up in court. https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title18-section2258A&num=0&edition=prelim https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim...
- nicce 5y ago
  > Apple isn't required to stop redistribution of existing material; they're required to make a report if they have actual knowledge of users possessing or distributing apparent CSAM. This isn't that simple. If NCMEC comes with the properties of CSAM (e.g. hashes) and asks provider especially those to be removed from their cloud, it is hard to remove them without looking for them. This is different than an obligation to actively look for CSAM in general.
  - Zak 5y ago
    Can you cite a statute that requires a provider to look for hashes when NCMEC asks them to? If NCMEC told a provider that a specific URL (or similarly unique identifier) contains CSAM, the provider would be obligated to destroy the associated file or be guilty of possession/distribution because at that point they know what they have. That's different from NCMEC providing hashes that could identify files the provider may or may not be storing.