6 ms·

Anyone who engaged in such a discussion with a non-technical person is going to defacto seem like an advocate for child pornography, similar to how advocating f

Anyone who engaged in such a discussion with a non-technical person is going to defacto seem like an advocate for child pornography, similar to how advocating for encryption can easily be twisted to being pro-crime. Having said that, there is an enormous amount of misinformation and fear-mongering about a pretty tame change. This seems like so much ado about very close to nothing. a) They optionally scan messaged photos for nudity using a NN if the participants are children and in a family (the account grouping), and the group adult(s) have opted in, giving children warnings and information if they send or receive such material. A+++ thumbs up. b) They scan photos that you've uploaded to iCloud (available at photos.icloud.com, unencrypted -- in the E2E sense, effectively "plaintext" from Apple's perspective -- etc) for known CP hashes. Bizarrely Apple decided to scan these on device as well, causing 99% of the outrage and confusion, yet every major cloud photo service in the world does such checks for the same reason, whether you have the photo set to private or not, and presumably Apple decided to do it on device simply as free distributed computing, taking advantage of hundreds of millions of high performance chips, but most importantly as a PR move demonstrating that "Apple Silicon helps with Child Safety", etc. That's it. Various "this is a harbinger of doom and tomorrow they're going to..." arguments are unconvincing. This does absolutely nothing to break or subvert E2E encryption or on device privacy. EDIT: The moderation of this comment has been fascinating, going to double digits, down to negatives, back up again, etc.

new299 5y ago
They were already doing it on the cloud: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... So now they’re doing it on device too. This feels like it’s putting in place the foundation to scan all offline content.
- defaultname 5y ago
  Scanning on device (albeit only of photos shared off device) seems like an ill-considered PR move for a whole child safety push (perhaps with a "look at how powerful our iPhone chips are" angle). As you mentioned, they've already been doing these checks for some time on their servers, and people concerned about false positives should realize that Microsoft, Google, Facebook, Amazon et al are doing identical checks with a very similar process. I imagine there are some frantic meetings at Apple today. However the grossly misleading claims people have been making to fear-monger aren't helpful.
- cwizou 5y ago
  Thanks for the link, I had assumed that Apple was already doing it on servers (like all other online services providers), which makes the announcement even more terrible. Moving it on device will show 0 improvement to the original goal, while opening a door that quite frankly I never expected Apple to be the one to open (I would have bet on Microsoft).
  - deleted 5y ago
    [deleted]
  - daemoon 5y ago
    > Moving it on device will show 0 improvement to the original goal, while opening a door that quite frankly I never expected Apple to be the one to open (I would have bet on Microsoft). The CSAM scan is only for photos that are to be uploaded to iCloud Photos. Turning off iCloud Photos will disable this.
    - cwizou 5y ago
      Sorry if my point wasn't clear, I do understand this yes. My point is that to my knowledge, this is the first time that an on device "content check" is being done (even if it's just for photos that will end up in iCloud). This is the precedent (the on device check) that makes me and some others uneasy, as pointed out in the linked letter. The fact that it applies only to photos going to the cloud is an implementation detail of the demonstrated technology. Legislators around the world now have a precedent and may (legitimately) want it extended to comply with their existing or upcoming laws. This is not a particularly far fetched scenario if you consider that Apple has already accommodated how they run their services locally (as they should, they have to comply with local laws around the world in order to be able to operate). That's the crux of the issue most of the people quoted in the letter have, one can argue it's just a slippery slope argument, I personally think that one can be legitimately concerned of the precedent being set. Keeping doing it on server, in my opinion, was a much better option for users (with the same compliance to local laws and effectiveness to the stated goal as far as we know, there's no improvement on that front, or none that couldn't have been brought to the existing server check), and ultimately also a safer option in the long run for Apple. They've opened themselves, for little reason, to a large amount of trouble on an international scale and at this point rolling it back (to server checks) might not make a difference anyway.
deleted 5y ago
[deleted]
roenxi 5y ago
"They're going to scan your phone and probably have someone review any photos with a lot of human flesh in them" would be enough to get a lot of non-technical users to take notice. That would get a lot of people nervous. Let alone anyone smart who thinks through the implications here of how far the line is being pushed on how public your phone is.
- davidcbc 5y ago
  It would, but luckily that's not what's happening
  - roenxi 5y ago
    Go read the announcement, the "CSAM detection" heading [0]. It is exactly what they are doing. Although they're assuring us that they don't make mistakes. The technical term for that is either going to be "blatant deception" or "delusion". Apple are impressive but they haven't developed a tech that can't make mistakes. [0] https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
    - defaultname 5y ago
      That isn't what they're doing at all. You have significantly misunderstood or conflated different sections. Though I don't blame you at all: Read through the various hysterical posts about this and there are a lot of extraordinary misrepresentations.
      - roenxi 5y ago
        Ah, I see what you're getting at. They're currently hashing for specific photos. I don't care. There is no way on this good earth that law enforcement is going to let them get away with that. They're claiming that they will be scanning things that are obviously child porn and ignoring it. That isn't a long term stable thing to be doing - if they think scanning for anything is ok there is no logical reason to stop here. So they probably aren't going to stop, and they certainly aren't going to announce every step they take to increase the net. And their 1:1,000,000,000,000 number is still delusional. The system is going to produce false positives. There are more sources of error here than the cryptographic hash algorithm.
schnable 5y ago
Apple does a lot of the ML and personalization stuff on user devices for privacy reasons as well, keeping your data out of the cloud, and that is a good thing.
syshum 5y ago
I think you are massively under estimating what this change means if you think it is "pretty tame change" Clearly you do not understand the full ramification of what is happening here.
- defaultname 5y ago
  What are the ramifications that I "do not understand"? I will repeat: It is a very tame change. Were you frantic and delirious when a neural network first identified a dog in your photos? Isn't that the slippery slope to it reporting you to the authorities for something bong shaped? Speaking of which, every bit of fear mongering relies upon a slippery slope fallacy. What is clearly a PR move is somehow actually the machinations of a massive surveillance network. Why? Why would Apple do that?
  - evrydayhustling 5y ago
    > Why would Apple do that? Because it gets required to by the laws of countries responsible for most of their market? And because authoritarian regimes intentionally blur the lines between criminal and political surveillance over time, making it harder for companies to draw hard policy lines? Concern about this doesn't require any bond villains, it just requires well-intentioned pragmatists on one side and idealogical politicos on the other. FWIW, I think you have a point about the doom-saying. Countries with good judicial protections around privacy already use it as a backstop against dirty tricks where folks use one type of surveillance to require another. But it makes sense to wonder how those barriers will erode over time, and to worry about places where they don't exist.
    - dwaite 5y ago
      In which case how is this a slippery slope? They didn’t do something and there was no legal mandate. Now they are required to do something to be able to operate in said company. Is the slippery slope that they can be in compliance faster?
  - robertlagrant 5y ago
noasaservice 5y ago
This on-device scanning is even worse. They cant even tell what was violating, or what hash, or what image. Just that the computer said you are violating. We have no idea about the hash collisions. When talking about whole world, 2^256 isn't a big enough space.... even if they're using 256 bits. And how dare anybody criticize this - criticism is tantamount to being for child porn. (Then again, that's why it was chosen. We'll soon see other things 'forbidden'.)
- defaultname 5y ago
  There is a shared (among all of the major tech companies and presumably law enforcement) hash database of child abuse material. Going from a photo to the hash is deterministic: How it gets to a hash on your phone is surely the same way it gets to a hash running the exact same algorithm on the cloud, whether iCloud, Amazon Photos, etc. Such a collision would generate a human validation. This applies to cloud-shared files. It actually has applied to cloud shared photos for years. It applies to literally every major cloud photo service.
  - noasaservice 5y ago
    1. how many false positives have there been? 2. is it really reviewed by a human? I see how YT works, and automated failure at scale is the name of the game 3. apple has said that the on-device scanning only provides a binary yes/no on CP detection. How do you defend against a "yes" accusation? (when stored on someone else's server, the evidence is there)
    - dwaite 5y ago
      This is part of iCloud photo sync, and on hitting some threshold of matching pictures it would trigger human review. There would also presumably be human review involved in the legal process, e.g. law enforcement getting a subpoena based on Apple notifying them, and then using gathered evidence for a warrant. The system is based on known image hashes, not arbitrary ML detection. As this system is used only for iCloud photo uploads, the evidence gathering should be similar to that done by LE with other cloud hosting providers for years
stephen_g 5y ago
I guess the upside might be that this could be a compromise for them to start doing end to end encryption on iCloud backups and iCloud photo libraries. They might be able to argue that if they’re scanning for illegal content on the client side, then they don’t need to be able to decrypt on the cloud side… We’ll have to see, it’s still creepy but potentially a small net gain overall if that becomes an option…
- tomjen3 5y ago
  If they claim e2e, but we scan on the client side then we should sue them for deceptive marketing. e2e means something specific and should be an absolute requirement in a post Snowdon age, not something that is optional or a compromise.
bambax 5y ago
> a) They scan photos for nudity using a NN if the participants are children and in a family (the account grouping), giving children warnings and information if they send or receive such material. A+++ thumbs up. Leave my kids alone. If they want to share photos of themselves naked, it's none of anyone's business except them and maybe me (maybe), certainly not a huge American corporation. Neither me or my kids have iPhones, but as others have observed, I have no illusions that Google will follow suit. Our options are becoming pretty limited at this point.
- barkerja 5y ago
  Hopefully this is another configurable option that falls under the already very extensive family screen time feature. I understand where you're coming from and respect your position, but I fall on the opposite side. This is something I do want for my kid.
  - defaultname 5y ago
    On the Child Safety page one of the dialogs is the opt in (or out) configuration, so it seems, as one would expect, that the adult(s) in the family sharing group get to configure this. And it's a useful, valuable option that many (I would wager the overwhelming majority) parents will enable. Apple made a huge PR mistake announcing both of these systems together (the CP hashing system and the NN message warning system), because as seen throughout the comments it has led to lots of people conflating and mixing and matching elements of both into a fearsome frankensystems.
    - wizzwizz4 5y ago
      The NN is the system I thought they were making, and I applaud it. The hashing one feels really dangerous, though; I don't think that's people just exaggerating. Apple hasn't done enough to limit their own power, so they might (read: will) be made to use it to hurt people.
    - zepto 5y ago
      > it has led to lots of people conflating and mixing and matching elements of both into a fearsome frankensystems. I agree they could have slowly walked people through the components one by one, but so much of what is in the comments is pure bad faith that I am not sure that it would have helped. By “bad faith”, I mean strongly asserting as true claims that people know they haven’t checked, and which later turn out to be false. In a cynical way this might actually be better PR for Apple. They know they can’t prevent people who dislike them from jumping to the most negative conclusions possible. By presenting these features together and letting the crazy interpretations abound, they make their opponents seem unhinged, and this obscures the real but less apocalyptic concerns.
tomjen3 5y ago
The problem with this is that they can now scan images on phones regardless of upload, and it will render any future promisses of e2e iCloud deceiving. I don't see a reason to do this, other than to either scan all images, or claim that iCloud is e2e in the future. And of course they went with the protection of children argument, which is bullshit. Apple gets paid by its users and should have no other interests than to get paid as much money as possible, regardless of who pays them.
- zepto 5y ago
  > Apple gets paid by its users and should have no other interests than to get paid as much money as possible, regardless of who pays them. That’s exactly why they are doing this. Providing a safe haven for child sex predators is bad for their brand.
silverpepsi 5y ago
Why does everyone mention they already did it on cloud like that has any relevance whatsoever? I have never once in my life thought about activating an automatic back up to cloud feature on any phone I have ever owned, for a single second. So yes, it is hella different. This is for all the same reasons I backup personal data only to my NAS and use cloud accounts for generic shit like purchased music backups and nothing more. I prefer losing all my photos if my phone is pickpocketed in between backups to having a public record of everything I ever photographed. Am I the 0.00000001% or something? I didn't even realize I was the odd man out, honestly.
- defaultname 5y ago
  "Why does everyone mention they already did it on cloud like that has any relevance whatsoever?" Given that this only applies to photos that are stored to the cloud, it seems like it has total relevance given that for users literally nothing has changed. To argue that there is some fearsome new development requires one to extrapolate into "Well what if..." arguments.
defaultname 5y ago
As a followup, to be clear on the intentions in my post, while I do think that a lot of the reactions have been over the top (there are numerous comments claiming outright falseshoods about this system, out of either ignorance or to prejudice), that Apple decided to do the CSAM stuff on device is incredibly ill considered. Do it in the cloud just like every other service does. None of this anger would have happened if they just kept it in the cloud, and I truly can not fathom how this made it this far. I would peg overwhelming odds that they abandon the on device idea as it makes no sense and has brought incredible ill will.

new299 5y ago

They were already doing it on the cloud: https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-icloud-photos-for-child-abuse-images/ https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-... So now they’re doing it on device too. This feels like it’s putting in place the foundation to scan all offline content.

defaultname 5y ago
Scanning on device (albeit only of photos shared off device) seems like an ill-considered PR move for a whole child safety push (perhaps with a "look at how powerful our iPhone chips are" angle). As you mentioned, they've already been doing these checks for some time on their servers, and people concerned about false positives should realize that Microsoft, Google, Facebook, Amazon et al are doing identical checks with a very similar process. I imagine there are some frantic meetings at Apple today. However the grossly misleading claims people have been making to fear-monger aren't helpful.
cwizou 5y ago
Thanks for the link, I had assumed that Apple was already doing it on servers (like all other online services providers), which makes the announcement even more terrible. Moving it on device will show 0 improvement to the original goal, while opening a door that quite frankly I never expected Apple to be the one to open (I would have bet on Microsoft).
- deleted 5y ago
  [deleted]
- daemoon 5y ago
  > Moving it on device will show 0 improvement to the original goal, while opening a door that quite frankly I never expected Apple to be the one to open (I would have bet on Microsoft). The CSAM scan is only for photos that are to be uploaded to iCloud Photos. Turning off iCloud Photos will disable this.
  - cwizou 5y ago
    Sorry if my point wasn't clear, I do understand this yes. My point is that to my knowledge, this is the first time that an on device "content check" is being done (even if it's just for photos that will end up in iCloud). This is the precedent (the on device check) that makes me and some others uneasy, as pointed out in the linked letter. The fact that it applies only to photos going to the cloud is an implementation detail of the demonstrated technology. Legislators around the world now have a precedent and may (legitimately) want it extended to comply with their existing or upcoming laws. This is not a particularly far fetched scenario if you consider that Apple has already accommodated how they run their services locally (as they should, they have to comply with local laws around the world in order to be able to operate). That's the crux of the issue most of the people quoted in the letter have, one can argue it's just a slippery slope argument, I personally think that one can be legitimately concerned of the precedent being set. Keeping doing it on server, in my opinion, was a much better option for users (with the same compliance to local laws and effectiveness to the stated goal as far as we know, there's no improvement on that front, or none that couldn't have been brought to the existing server check), and ultimately also a safer option in the long run for Apple. They've opened themselves, for little reason, to a large amount of trouble on an international scale and at this point rolling it back (to server checks) might not make a difference anyway.

deleted 5y ago

[deleted]

roenxi 5y ago

"They're going to scan your phone and probably have someone review any photos with a lot of human flesh in them" would be enough to get a lot of non-technical users to take notice. That would get a lot of people nervous. Let alone anyone smart who thinks through the implications here of how far the line is being pushed on how public your phone is.

davidcbc 5y ago
It would, but luckily that's not what's happening
- roenxi 5y ago
  Go read the announcement, the "CSAM detection" heading [0]. It is exactly what they are doing. Although they're assuring us that they don't make mistakes. The technical term for that is either going to be "blatant deception" or "delusion". Apple are impressive but they haven't developed a tech that can't make mistakes. [0] https://www.apple.com/child-safety/ https://www.apple.com/child-safety/
  - defaultname 5y ago
    That isn't what they're doing at all. You have significantly misunderstood or conflated different sections. Though I don't blame you at all: Read through the various hysterical posts about this and there are a lot of extraordinary misrepresentations.
    - roenxi 5y ago
      Ah, I see what you're getting at. They're currently hashing for specific photos. I don't care. There is no way on this good earth that law enforcement is going to let them get away with that. They're claiming that they will be scanning things that are obviously child porn and ignoring it. That isn't a long term stable thing to be doing - if they think scanning for anything is ok there is no logical reason to stop here. So they probably aren't going to stop, and they certainly aren't going to announce every step they take to increase the net. And their 1:1,000,000,000,000 number is still delusional. The system is going to produce false positives. There are more sources of error here than the cryptographic hash algorithm.

schnable 5y ago

Apple does a lot of the ML and personalization stuff on user devices for privacy reasons as well, keeping your data out of the cloud, and that is a good thing.

syshum 5y ago

I think you are massively under estimating what this change means if you think it is "pretty tame change" Clearly you do not understand the full ramification of what is happening here.

defaultname 5y ago
What are the ramifications that I "do not understand"? I will repeat: It is a very tame change. Were you frantic and delirious when a neural network first identified a dog in your photos? Isn't that the slippery slope to it reporting you to the authorities for something bong shaped? Speaking of which, every bit of fear mongering relies upon a slippery slope fallacy. What is clearly a PR move is somehow actually the machinations of a massive surveillance network. Why? Why would Apple do that?
- evrydayhustling 5y ago
  > Why would Apple do that? Because it gets required to by the laws of countries responsible for most of their market? And because authoritarian regimes intentionally blur the lines between criminal and political surveillance over time, making it harder for companies to draw hard policy lines? Concern about this doesn't require any bond villains, it just requires well-intentioned pragmatists on one side and idealogical politicos on the other. FWIW, I think you have a point about the doom-saying. Countries with good judicial protections around privacy already use it as a backstop against dirty tricks where folks use one type of surveillance to require another. But it makes sense to wonder how those barriers will erode over time, and to worry about places where they don't exist.
  - dwaite 5y ago
    In which case how is this a slippery slope? They didn’t do something and there was no legal mandate. Now they are required to do something to be able to operate in said company. Is the slippery slope that they can be in compliance faster?
- robertlagrant 5y ago

noasaservice 5y ago

This on-device scanning is even worse. They cant even tell what was violating, or what hash, or what image. Just that the computer said you are violating. We have no idea about the hash collisions. When talking about whole world, 2^256 isn't a big enough space.... even if they're using 256 bits. And how dare anybody criticize this - criticism is tantamount to being for child porn. (Then again, that's why it was chosen. We'll soon see other things 'forbidden'.)

defaultname 5y ago
There is a shared (among all of the major tech companies and presumably law enforcement) hash database of child abuse material. Going from a photo to the hash is deterministic: How it gets to a hash on your phone is surely the same way it gets to a hash running the exact same algorithm on the cloud, whether iCloud, Amazon Photos, etc. Such a collision would generate a human validation. This applies to cloud-shared files. It actually has applied to cloud shared photos for years. It applies to literally every major cloud photo service.
- noasaservice 5y ago
  1. how many false positives have there been? 2. is it really reviewed by a human? I see how YT works, and automated failure at scale is the name of the game 3. apple has said that the on-device scanning only provides a binary yes/no on CP detection. How do you defend against a "yes" accusation? (when stored on someone else's server, the evidence is there)
  - dwaite 5y ago
    This is part of iCloud photo sync, and on hitting some threshold of matching pictures it would trigger human review. There would also presumably be human review involved in the legal process, e.g. law enforcement getting a subpoena based on Apple notifying them, and then using gathered evidence for a warrant. The system is based on known image hashes, not arbitrary ML detection. As this system is used only for iCloud photo uploads, the evidence gathering should be similar to that done by LE with other cloud hosting providers for years

stephen_g 5y ago

I guess the upside might be that this could be a compromise for them to start doing end to end encryption on iCloud backups and iCloud photo libraries. They might be able to argue that if they’re scanning for illegal content on the client side, then they don’t need to be able to decrypt on the cloud side… We’ll have to see, it’s still creepy but potentially a small net gain overall if that becomes an option…

tomjen3 5y ago
If they claim e2e, but we scan on the client side then we should sue them for deceptive marketing. e2e means something specific and should be an absolute requirement in a post Snowdon age, not something that is optional or a compromise.

bambax 5y ago

> a) They scan photos for nudity using a NN if the participants are children and in a family (the account grouping), giving children warnings and information if they send or receive such material. A+++ thumbs up. Leave my kids alone. If they want to share photos of themselves naked, it's none of anyone's business except them and maybe me (maybe), certainly not a huge American corporation. Neither me or my kids have iPhones, but as others have observed, I have no illusions that Google will follow suit. Our options are becoming pretty limited at this point.

barkerja 5y ago
Hopefully this is another configurable option that falls under the already very extensive family screen time feature. I understand where you're coming from and respect your position, but I fall on the opposite side. This is something I do want for my kid.
- defaultname 5y ago
  On the Child Safety page one of the dialogs is the opt in (or out) configuration, so it seems, as one would expect, that the adult(s) in the family sharing group get to configure this. And it's a useful, valuable option that many (I would wager the overwhelming majority) parents will enable. Apple made a huge PR mistake announcing both of these systems together (the CP hashing system and the NN message warning system), because as seen throughout the comments it has led to lots of people conflating and mixing and matching elements of both into a fearsome frankensystems.
  - wizzwizz4 5y ago
    The NN is the system I thought they were making, and I applaud it. The hashing one feels really dangerous, though; I don't think that's people just exaggerating. Apple hasn't done enough to limit their own power, so they might (read: will) be made to use it to hurt people.
  - zepto 5y ago
    > it has led to lots of people conflating and mixing and matching elements of both into a fearsome frankensystems. I agree they could have slowly walked people through the components one by one, but so much of what is in the comments is pure bad faith that I am not sure that it would have helped. By “bad faith”, I mean strongly asserting as true claims that people know they haven’t checked, and which later turn out to be false. In a cynical way this might actually be better PR for Apple. They know they can’t prevent people who dislike them from jumping to the most negative conclusions possible. By presenting these features together and letting the crazy interpretations abound, they make their opponents seem unhinged, and this obscures the real but less apocalyptic concerns.

tomjen3 5y ago

The problem with this is that they can now scan images on phones regardless of upload, and it will render any future promisses of e2e iCloud deceiving. I don't see a reason to do this, other than to either scan all images, or claim that iCloud is e2e in the future. And of course they went with the protection of children argument, which is bullshit. Apple gets paid by its users and should have no other interests than to get paid as much money as possible, regardless of who pays them.

zepto 5y ago
> Apple gets paid by its users and should have no other interests than to get paid as much money as possible, regardless of who pays them. That’s exactly why they are doing this. Providing a safe haven for child sex predators is bad for their brand.

silverpepsi 5y ago

Why does everyone mention they already did it on cloud like that has any relevance whatsoever? I have never once in my life thought about activating an automatic back up to cloud feature on any phone I have ever owned, for a single second. So yes, it is hella different. This is for all the same reasons I backup personal data only to my NAS and use cloud accounts for generic shit like purchased music backups and nothing more. I prefer losing all my photos if my phone is pickpocketed in between backups to having a public record of everything I ever photographed. Am I the 0.00000001% or something? I didn't even realize I was the odd man out, honestly.

defaultname 5y ago
"Why does everyone mention they already did it on cloud like that has any relevance whatsoever?" Given that this only applies to photos that are stored to the cloud, it seems like it has total relevance given that for users literally nothing has changed. To argue that there is some fearsome new development requires one to extrapolate into "Well what if..." arguments.

defaultname 5y ago

As a followup, to be clear on the intentions in my post, while I do think that a lot of the reactions have been over the top (there are numerous comments claiming outright falseshoods about this system, out of either ignorance or to prejudice), that Apple decided to do the CSAM stuff on device is incredibly ill considered. Do it in the cloud just like every other service does. None of this anger would have happened if they just kept it in the cloud, and I truly can not fathom how this made it this far. I would peg overwhelming odds that they abandon the on device idea as it makes no sense and has brought incredible ill will.