5 ms·

> As we’ve explained in Deeplinks blog posts, Apple’s planned phone-scanning system opens the door to broader abuses. It decreases privacy for all iCloud photo

> As we’ve explained in Deeplinks blog posts, Apple’s planned phone-scanning system opens the door to broader abuses. It decreases privacy for all iCloud photo users, and the parental notification system is a shift away from strong end-to-end encryption. It will tempt liberal democratic regimes to increase surveillance, and likely bring even great pressures from regimes that already have online censorship ensconced in law. What’s clear is the potential for future abuse mandated by various gov (though that could easily be the same even without all these CSAM measures.. not a new threat). However, the other things are erroneously lumped in with it. It only weakens privacy for iCloud photos if you have CP or photos in a CP database, and it doesn’t prevent E2E with texting… it just gives an additional parental control in addition to the many numerous parental controls (with them kids have no privacy already), and is totally unrelated to the CSAM effort. I admire the EFF in many ways, but I wish they’d be more exact here given their access to expert knowledge.

kurthr 5y ago
"it just gives an additional parental control in addition to the many numerous parental controls (with them kids have no privacy already)" Wait, sending data (of matching CP hashes) to law enforcement is parental control?
- aesh2Xa1 5y ago
  Yours is a very fair negative reaction. The information in the EFF’s includes a portion where it seems to be concerned only about alerting parents[1]. I think many parents would find that reasonable. However, the fact that the information will also be sent to the government [2] is just plainly an abuse of privacy, goes outside of the relationship between parent and child, and I do not imagine that parents would find that reasonable. > [1] Moreover, the system Apple has developed assumes that the "parent" and "child" accounts involved actually belong to an adult who is the parent of a child, and that those individuals have a healthy relationship. This may not always be the case; an abusive adult may be the organiser of the account, and the consequences of parental notification could threaten the child’s safety and wellbeing. LGBTQ+ youths on family accounts with unsympathetic parents are particularly at risk. As a result of this change, iMessages will no longer provide confidentiality and privacy to those users through an end-to-end encrypted messaging system in which only the sender and intended recipients have access to the information sent. > [2] When a preset threshold number of matches is met, it will disable the account and report the user and those images to authorities.
  - snowwrestler 5y ago
    Your [1] and [2] refer to separate systems. The parental control does not send info to authorities.
- snowwrestler 5y ago
  Apple announced two separate things in one press release: a CSAM-scanning system, and a parental control that uses AI to attempt to detect nude pictures in iMessages and alert the parents. The latter system does not send any info to Apple or any authorities.
carlosdp 5y ago
> (though that could easily be the same even without all these CSAM measures.. not a new threat) Apple has historically avoided being pressured by governments to allow this kind of surveillance by arguing they can't be forced to create functionality that doesn't exist, or hand over information they don't have. It's the argument they made in the San Bernadino case. If they release this, it'll be much harder to avoid giving governments that existing ability for other, non-CSAM uses.
- shuckles 5y ago
  The request from the FBI in the San Bernardino case was to change a passcode limit constant and retry timeouts. Those are about as trivial to implement as any of the convoluted government coercion database attacks against CSAM detection being proposed here.
  - ajsnigrutin 5y ago
    The difference here is, that apple would have to develop a new feature for them, test it, and waste millions in lawers to protect themselves from accusations of tampering with the evidence (which a software update definitely is, and who knows what FBI wanted in that software update, maybe even to insert a fake sms to the sms database, or many other things a good defense lawyer could bring to the jury). Here, it's different.. let's say there's a new wikileaks, photos of secret documents... and again, first a few journalists get the data, start slowly writing articles, and FBI just adds the hashes to the database, and they can find out who has the photos, with metadata even who had the first, before the journalist, and they can find a leak.
    - azinman2 5y ago
      But the FBI can’t just add the hashes to the db. That’s why it’s the intersection of two dbs in two jurisdictions… to prevent exactly that kind of attack. Then they need to pass a human reviewer as well.
      - xkcd-sucks 5y ago
jtbayly 5y ago
> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Or people who have photos that hash the same as CP.
- smoldesu 5y ago
  Or possess tampered photos that were engineered to be a hash collision.
  - azinman2 5y ago
    You’d have to not only have over 30 hash collisions, but also have it collide with another secret hash function, and then also have a human look at it and agree it’s CP. So what’s the actual realistic issue here? This keeps getting thrown around as if it’s likely, yet not only are there numerous steps against this in the Apple chain, this would already be a huge issue with Dropbox, Facebook, Microsoft, Google, etc who do CP scanning according to all of the comments on HN.
    - nullc 5y ago
      > You’d have to not only have over 30 hash collisions That's trivial. If the attacker can get one image onto your device they can get several. It's very easy to construct preimages for Apple's neural hash function, including fairly good looking ones (e.g. https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-902977931 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... ) > collide with another secret hash function The supposed other 'secret' hash function cannot be secret from the state actors generating the databases. Also, if it has a similar structure/training, it's not that unlikely that the same images would collide by chance. > also have a human look at it and agree it’s CP That's straight-forward: simply use nude or pornographic images which looks like they could be children or ones where without context you can't tell. It's a felony for them to fail to report child porn if they see it in review, and the NCMEC guidance tells people when in doubt to report. Besides, once someone else has looked at your pictures your privacy has been violated.
relax88 5y ago
> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Who controls what is in the database? What independent oversight ensures that it’s only CSAM images? The public certainly can’t audit it. What is stopping the CCP from putting pro-Uighur or Xi Winnie the Pooh images into the database? Or from the US using this to locate images that are interesting from an intelligence perspective (Say for example pictures of Iranian Uranium centrifuges)? Apple says they will only add images that more than one government request? All it would take is a few G men to show up at Apple with a secret court order to do this no? So… China and Hong Kong? The Five Eyes nations?
- nullc 5y ago
  Their use of a highly vulnerable[1] "neural" perceptual hash function makes the database unauditable: An abusive state actor could obtain child porn images and invisibly alter them to match the hashes of the ideological or ethnically related images they really want to match. If challenged, they could produce child porn images matching their database, and they could had these images to other governments to unknowingly or plausibly denyably include. ...but they don't have to do anything that elaborate because Apple is using powerful cryptography against their users to protect themselves and their data sources from any accountability for the content of the database: The hashes in the database are hidden from everyone who isn't Apple or a state agent. There is no opportunity to learn, much less challenge the content of the database. [1] https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
  - azinman2 5y ago
    They have to come from the intersection of two databases from two jurisdictions. So already that’s out as you suggest. Then you’d have to match _nearly exact photos_, which isn’t a vector for general photos of some random minority. Then you’d need 30 of such specific photos, a match with another secret hash, and then a human reviewer at Apple has to say yes it’s CP before anything else happens. I think there are plenty of reasons to be concerned about future laws and future implementations, but let’s be honest about the real risks of this today as it’s currently implemented.
sennight 5y ago
> ...if you have CP or photos in a CP database... Which database? I get the impression that people think there is a singular repository for thoroughly vetted and highly controlled CP evidence submission. No such thing exists.
- azinman2 5y ago
  It’s an intersection between a US db and a not yet chosen non-US db, which then will have a human reviewer verify its CP before sending off to the authorities.
  - sennight 5y ago
    > What more could one ask for? An independent audit for both the secret secondary perceptual hashing algorithm and the chain of custody policies/compliance for the "US db" and the disconcertedly open ended "not yet chosen non-US db"?
    - azinman2 5y ago
      Do you ask for the same audit at Facebook, Google, Microsoft, Dropbox, and countless others who are already doing this and have been for years? I do not share your same concern of some abused db _today_.
      - amadeuspagel 5y ago
        Neither Google nor Microsoft scan pictures people have on their devices running Android or Windows. I'm not sure how that's even applicable to Facebook and Dropbox.
        azinman2 5y ago
        You’re asking for an auditing chain presumably due to concerns about governments putting in photos of things that aren’t CP. Apple is only doing this for photos that get uploaded to iCloud, with this neural hash that gets uploaded with it. The actual verification of a CP match occurs on the server due to the hashes being blinded. So in many ways, it’s very similar to what these other cloud providers effectively do — search for CP matches on uploaded data. If you’re concerned about non-CP being scanned for, then you should already be concerned about that with everyone else. Thus if you’re asking for auditing of Apple, then you should widen your request. If you do, then sure, I can understand that. If you’re not concerned about the existing system, then I think you’re being non-consistent. Most people in comments to me seem to be non-consistent, and are being overly knee jerk about this… myself included initially.
lamontcg 5y ago
> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Or if someone hacks your device and uploads one of those photos to your iCloud. (I still have no idea why people aren't pointing this out more aggressively -- phones get hacked probably every minute of every day, and acquiring those actual photos isn't that difficult once you're willing to commit felonies -- hash collisions are a distraction)
- azinman2 5y ago
  Because that then would already be a problem for Facebook, Google, Microsoft, etc that host photos that hacked phones could be uploading today. And we’re just not seeing that being the case. Because all these providers have been doing this for so many years, including the nearly 17 million photos identified by Facebook last year, you’d figure there would be a lot more noise if this was really going on. In fact, I would venture to say it’s far easier to hack a Facebook account than it is iCloud that has many on-device-based security protections that a cloud login without mandatory 2FA (and often SMS when it is used).
  - lamontcg 5y ago
    We had SWAT teams for a long time before SWATing became popular. The publicity that this has gotten is only going to increase the chances that all these services start getting abused. And who is to say that it hasn't happened already and been entirely successful, but nobody believed the victim.
    - azinman2 5y ago
      So you think we’re going to see a rise in people uploading CP to others cloud providers?
      - lamontcg 5y ago
        Yes. I would bet the large majority of people here who are now quick to point out that other cloud providers have been doing this for years didn't know that fact a month ago. We're now well armed with that information due to arguing about this Apple issue. The fact that you're so quick to inform me of the facts is precisely why I think its more likely to happen.

kurthr 5y ago

"it just gives an additional parental control in addition to the many numerous parental controls (with them kids have no privacy already)" Wait, sending data (of matching CP hashes) to law enforcement is parental control?

aesh2Xa1 5y ago
Yours is a very fair negative reaction. The information in the EFF’s includes a portion where it seems to be concerned only about alerting parents[1]. I think many parents would find that reasonable. However, the fact that the information will also be sent to the government [2] is just plainly an abuse of privacy, goes outside of the relationship between parent and child, and I do not imagine that parents would find that reasonable. > [1] Moreover, the system Apple has developed assumes that the "parent" and "child" accounts involved actually belong to an adult who is the parent of a child, and that those individuals have a healthy relationship. This may not always be the case; an abusive adult may be the organiser of the account, and the consequences of parental notification could threaten the child’s safety and wellbeing. LGBTQ+ youths on family accounts with unsympathetic parents are particularly at risk. As a result of this change, iMessages will no longer provide confidentiality and privacy to those users through an end-to-end encrypted messaging system in which only the sender and intended recipients have access to the information sent. > [2] When a preset threshold number of matches is met, it will disable the account and report the user and those images to authorities.
- snowwrestler 5y ago
  Your [1] and [2] refer to separate systems. The parental control does not send info to authorities.
snowwrestler 5y ago
Apple announced two separate things in one press release: a CSAM-scanning system, and a parental control that uses AI to attempt to detect nude pictures in iMessages and alert the parents. The latter system does not send any info to Apple or any authorities.

carlosdp 5y ago

> (though that could easily be the same even without all these CSAM measures.. not a new threat) Apple has historically avoided being pressured by governments to allow this kind of surveillance by arguing they can't be forced to create functionality that doesn't exist, or hand over information they don't have. It's the argument they made in the San Bernadino case. If they release this, it'll be much harder to avoid giving governments that existing ability for other, non-CSAM uses.

shuckles 5y ago
The request from the FBI in the San Bernardino case was to change a passcode limit constant and retry timeouts. Those are about as trivial to implement as any of the convoluted government coercion database attacks against CSAM detection being proposed here.
- ajsnigrutin 5y ago
  The difference here is, that apple would have to develop a new feature for them, test it, and waste millions in lawers to protect themselves from accusations of tampering with the evidence (which a software update definitely is, and who knows what FBI wanted in that software update, maybe even to insert a fake sms to the sms database, or many other things a good defense lawyer could bring to the jury). Here, it's different.. let's say there's a new wikileaks, photos of secret documents... and again, first a few journalists get the data, start slowly writing articles, and FBI just adds the hashes to the database, and they can find out who has the photos, with metadata even who had the first, before the journalist, and they can find a leak.
  - azinman2 5y ago
    But the FBI can’t just add the hashes to the db. That’s why it’s the intersection of two dbs in two jurisdictions… to prevent exactly that kind of attack. Then they need to pass a human reviewer as well.
    - xkcd-sucks 5y ago

jtbayly 5y ago

> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Or people who have photos that hash the same as CP.

smoldesu 5y ago
Or possess tampered photos that were engineered to be a hash collision.
- azinman2 5y ago
  You’d have to not only have over 30 hash collisions, but also have it collide with another secret hash function, and then also have a human look at it and agree it’s CP. So what’s the actual realistic issue here? This keeps getting thrown around as if it’s likely, yet not only are there numerous steps against this in the Apple chain, this would already be a huge issue with Dropbox, Facebook, Microsoft, Google, etc who do CP scanning according to all of the comments on HN.
  - nullc 5y ago
    > You’d have to not only have over 30 hash collisions That's trivial. If the attacker can get one image onto your device they can get several. It's very easy to construct preimages for Apple's neural hash function, including fairly good looking ones (e.g. https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-902977931 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue... ) > collide with another secret hash function The supposed other 'secret' hash function cannot be secret from the state actors generating the databases. Also, if it has a similar structure/training, it's not that unlikely that the same images would collide by chance. > also have a human look at it and agree it’s CP That's straight-forward: simply use nude or pornographic images which looks like they could be children or ones where without context you can't tell. It's a felony for them to fail to report child porn if they see it in review, and the NCMEC guidance tells people when in doubt to report. Besides, once someone else has looked at your pictures your privacy has been violated.

relax88 5y ago

> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Who controls what is in the database? What independent oversight ensures that it’s only CSAM images? The public certainly can’t audit it. What is stopping the CCP from putting pro-Uighur or Xi Winnie the Pooh images into the database? Or from the US using this to locate images that are interesting from an intelligence perspective (Say for example pictures of Iranian Uranium centrifuges)? Apple says they will only add images that more than one government request? All it would take is a few G men to show up at Apple with a secret court order to do this no? So… China and Hong Kong? The Five Eyes nations?

nullc 5y ago
Their use of a highly vulnerable[1] "neural" perceptual hash function makes the database unauditable: An abusive state actor could obtain child porn images and invisibly alter them to match the hashes of the ideological or ethnically related images they really want to match. If challenged, they could produce child porn images matching their database, and they could had these images to other governments to unknowingly or plausibly denyably include. ...but they don't have to do anything that elaborate because Apple is using powerful cryptography against their users to protect themselves and their data sources from any accountability for the content of the database: The hashes in the database are hidden from everyone who isn't Apple or a state agent. There is no opportunity to learn, much less challenge the content of the database. [1] https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1#issuecomment-903094036 https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
- azinman2 5y ago
  They have to come from the intersection of two databases from two jurisdictions. So already that’s out as you suggest. Then you’d have to match _nearly exact photos_, which isn’t a vector for general photos of some random minority. Then you’d need 30 of such specific photos, a match with another secret hash, and then a human reviewer at Apple has to say yes it’s CP before anything else happens. I think there are plenty of reasons to be concerned about future laws and future implementations, but let’s be honest about the real risks of this today as it’s currently implemented.

sennight 5y ago

> ...if you have CP or photos in a CP database... Which database? I get the impression that people think there is a singular repository for thoroughly vetted and highly controlled CP evidence submission. No such thing exists.

azinman2 5y ago
It’s an intersection between a US db and a not yet chosen non-US db, which then will have a human reviewer verify its CP before sending off to the authorities.
- sennight 5y ago
  > What more could one ask for? An independent audit for both the secret secondary perceptual hashing algorithm and the chain of custody policies/compliance for the "US db" and the disconcertedly open ended "not yet chosen non-US db"?
  - azinman2 5y ago
    Do you ask for the same audit at Facebook, Google, Microsoft, Dropbox, and countless others who are already doing this and have been for years? I do not share your same concern of some abused db _today_.
    - amadeuspagel 5y ago
      Neither Google nor Microsoft scan pictures people have on their devices running Android or Windows. I'm not sure how that's even applicable to Facebook and Dropbox.
      - azinman2 5y ago
        You’re asking for an auditing chain presumably due to concerns about governments putting in photos of things that aren’t CP. Apple is only doing this for photos that get uploaded to iCloud, with this neural hash that gets uploaded with it. The actual verification of a CP match occurs on the server due to the hashes being blinded. So in many ways, it’s very similar to what these other cloud providers effectively do — search for CP matches on uploaded data. If you’re concerned about non-CP being scanned for, then you should already be concerned about that with everyone else. Thus if you’re asking for auditing of Apple, then you should widen your request. If you do, then sure, I can understand that. If you’re not concerned about the existing system, then I think you’re being non-consistent. Most people in comments to me seem to be non-consistent, and are being overly knee jerk about this… myself included initially.

lamontcg 5y ago

> It only weakens privacy for iCloud photos if you have CP or photos in a CP database Or if someone hacks your device and uploads one of those photos to your iCloud. (I still have no idea why people aren't pointing this out more aggressively -- phones get hacked probably every minute of every day, and acquiring those actual photos isn't that difficult once you're willing to commit felonies -- hash collisions are a distraction)

azinman2 5y ago
Because that then would already be a problem for Facebook, Google, Microsoft, etc that host photos that hacked phones could be uploading today. And we’re just not seeing that being the case. Because all these providers have been doing this for so many years, including the nearly 17 million photos identified by Facebook last year, you’d figure there would be a lot more noise if this was really going on. In fact, I would venture to say it’s far easier to hack a Facebook account than it is iCloud that has many on-device-based security protections that a cloud login without mandatory 2FA (and often SMS when it is used).
- lamontcg 5y ago
  We had SWAT teams for a long time before SWATing became popular. The publicity that this has gotten is only going to increase the chances that all these services start getting abused. And who is to say that it hasn't happened already and been entirely successful, but nobody believed the victim.
  - azinman2 5y ago
    So you think we’re going to see a rise in people uploading CP to others cloud providers?
    - lamontcg 5y ago
      Yes. I would bet the large majority of people here who are now quick to point out that other cloud providers have been doing this for years didn't know that fact a month ago. We're now well armed with that information due to arguing about this Apple issue. The fact that you're so quick to inform me of the facts is precisely why I think its more likely to happen.