5 ms·

The referenced Apple blog post[1] is pretty clear on what this feature does, and I wish the author at lapcatsoftware (as well as folks here) would have read it

The referenced Apple blog post[1] is pretty clear on what this feature does, and I wish the author at lapcatsoftware (as well as folks here) would have read it too, instead of taking the blog post as-is. Apple has implemented homomorphic encryption[2], which they can use to compute distance metrics such as cosine similarity without revealing the original query/embedding to the server. In the case of photos, an on-device model is first used to detect if a landmark may be possibly present in the image, and then sends an encrypted embedding[3] for the image region containing said landmark, which is evaluated on the server using HE techniques and then the encrypted results are sent back. I'm sure someone will come along and say Apple does none of these things; in which case, said commenter should probably not use Apple devices since, there is no reason to trust the toggles for "Enhanced Visual Search", and perhaps more importantly, the one for "Advanced Data Protection" either. However, I rarely see any other major company researching ML and HE together, so that alone is a sufficient positive indicator in my eyes. (It's interesting that this is also being downvoted. Please go ahead, I can't stop you :P but please also write the parts that you disagree with. Thank you.) [1] https://machinelearning.apple.com/research/homomorphic-encryption https://machinelearning.apple.com/research/homomorphic-encry... [2] https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption [3] https://mlr.cdn-apple.com/video/HE_Fig_3_PNNS_889f3a279c.m4v https://mlr.cdn-apple.com/video/HE_Fig_3_PNNS_889f3a279c.m4v

j45 2y ago
I thought less about the data, as much as the metadata, direct and indirect.
penguinburglar 2y ago
Thank you for posting the apple blog post. As usual, it's really interesting research, and it's fascinating to see how they solve potentially gnarly privacy issues.
- lapcat 2y ago
  > Thank you for posting the apple blog post. The very same blog post is linked in the submitted article.
  - deleted 2y ago
    [deleted]
gigel82 2y ago
That's not the point of the outrage though (at least not for me). They enabled by default a feature that analyzes my pictures (which I never upload to iCloud) and sends information about them to their (and others') servers. That is a gross violation of privacy. To be clear, I don't care about any encryption scheme they may be using, the gist is that they feel entitled to reach into their users' most private data (the photos they explicitly said they don't want to upload to iCloud) and "analyze" them. This is the same as that time Microsoft enabled OneDrive "by mistake" and started slurping people's private documents and photos saved in default locations (arguably worse since no one takes pictures with their PC's webcams).
- supriyo-biswas 2y ago
  If the data is encrypted, does the concern still apply? You bring up the example of Onedrive, but there is no use of e2e encryption or HE techniques there.
  - gigel82 2y ago
    Yes, of course, the concern is the data being exfiltrated to begin with. Like someone else in this thread mentioned, if they upload a single pixel from my image without my consent, that is too much data being uploaded without my consent.
    - scratchyone 2y ago
      If they sent a completely randomly generated integer from your phone without consent, would that be okay with you? Genuine question.
      - JTyQZSnP3cQGa8B 2y ago
        > If they sent [...] without consent, would that be okay with you? No. That would never be acceptable to me.
      - amiga386 2y ago
joe_the_user 2y ago
Just from memory when the scheme came up in earlier discussion. The system is essentially scanning for the signature for some known set of images of abuse so that it aims to capture abusers who would naively keep just these images on their machines. (It can't determine if a new image is abusive, notably). It's conceivable some number of (foolish and abusive) people will be caught this way and those favoring a long dragnet for this stuff will be happy. But this opens the possibility that a hacker could upload an image to an innocent person's computer and get that person arrested. Those favoring the long dragnet will naturally say the ends justify the means and you can't make an omelet without cracking a few eggs. Oh, "think of the children". Edit: Also worth adding that once a company is scanning user content to try to decide if the user is bad, it makes it that much easier to scan all kind of content in all kind of ways for all kinds of reasons. "for the good", naturally.
- Thorrez 2y ago
  >The system is essentially scanning for the signature for some known set of images of abuse Huh? The system is scanning for landmarks, not images of abuse. >people will be caught this way Due to the homomorphic encryption, I don't think Apple even knows whether the image matches a landmark in Apple's server database or not. So even if Apple put some images of abuse into its server database (which Apple claims only contains pictures of landmarks), I don't think Apple would know whether there was a match or not.
  - zmgsabst 2y ago
    Does Apple explicitly say that? Or only that they don’t know which landmark it matched?
    - scratchyone 2y ago
      Fundamentally, vector search like this doesn't have a concept of something "matching" or "not matching". It's just a cosine similarity value. To determine if an image "matches", you have to check if that similarity is within some given threshold. If the results of the cosine similarity operation are encrypted (they are with HE), that wouldn't be possible to determine. The bigger privacy risk would be that the device routes the request to a specific database shard based on whichever has a center-point closest to the image embedding on the device. They take steps to protect this information such as third-party proxying to hide user IP addresses, as well as having devices send fake requests so that the server cannot tell which are real user data and which are fake data.
charles_f 2y ago
> please also write the parts that you disagree with. Thank you The problem invoked by the article is that data is being sent back to Apple by default. Saying "it's fine because it's encrypted" and "don't use apple if you're not fine with that" doesn't help. The post complains about a product that stored sensitive customer content locally now sends that data to Apple, and given the combination of abuse on privacy and horrendous, generalized security failures that we've seen across the industry, those concerns seem genuine. Your comment is very dismissive of these concerns, which would explain why it's being down voted.
- dwaite 2y ago
  Apple also makes a mandatory API call to captive.apple.com from every device with a web view, just about every time they connect to a network. If someone is willing to take a hardline stance that a vendor should inform the user and require opt-in consent for every type of exchange, they are going to have to run platforms specifically targeted toward that mindset.
zaroth 2y ago
Everything about what they did was absolutely fantastic and amazing, until they turned it on by default.
ustad 2y ago
You're extensively describing the technical implementation while missing the fundamental issue: Why is Apple enabling this feature by default for what is, essentially, a luxury photo search feature? Let's break this down: 1. Nobody is questioning whether Apple's technical implementation is sophisticated or secure. It clearly is. 2. Nobody is suggesting the privacy toggles don't work. They do. 3. The issue is about Apple deciding that automatically sending data about users' photos (regardless of how securely) is an acceptable default for a feature many users may never want or need. Consider the value proposition here: Apple invested significant engineering resources into complex homomorphic encryption and differential privacy... so users can search for landmarks in their photos? And they deemed this feature so essential that it should be enabled by default? This feels like using a golden vault with military-grade security to store grocery lists. Yes, the security is impressive, but that's not the point. The point is: Why are my grocery lists being moved to the vault without asking me first? A privacy-respecting approach would be: "Hey, would you like to enable enhanced landmark search in your photos? We've built some really impressive privacy protections to make this secure..." Instead of: "We've already started analyzing your photos for landmarks because we built really impressive privacy protections..." The sophistication of the technology doesn't justify making it the default for what is ultimately an optional convenience feature.
- deleted 2y ago
  [deleted]
- slashdave 2y ago
  > users may never want or need Are you assuming that Apple did not perform a market analysis when implementing this feature? I think that is unlikely, considering the effort.

j45 2y ago

I thought less about the data, as much as the metadata, direct and indirect.

penguinburglar 2y ago

Thank you for posting the apple blog post. As usual, it's really interesting research, and it's fascinating to see how they solve potentially gnarly privacy issues.

lapcat 2y ago
> Thank you for posting the apple blog post. The very same blog post is linked in the submitted article.
- deleted 2y ago
  [deleted]

gigel82 2y ago

That's not the point of the outrage though (at least not for me). They enabled by default a feature that analyzes my pictures (which I never upload to iCloud) and sends information about them to their (and others') servers. That is a gross violation of privacy. To be clear, I don't care about any encryption scheme they may be using, the gist is that they feel entitled to reach into their users' most private data (the photos they explicitly said they don't want to upload to iCloud) and "analyze" them. This is the same as that time Microsoft enabled OneDrive "by mistake" and started slurping people's private documents and photos saved in default locations (arguably worse since no one takes pictures with their PC's webcams).

supriyo-biswas 2y ago
If the data is encrypted, does the concern still apply? You bring up the example of Onedrive, but there is no use of e2e encryption or HE techniques there.
- gigel82 2y ago
  Yes, of course, the concern is the data being exfiltrated to begin with. Like someone else in this thread mentioned, if they upload a single pixel from my image without my consent, that is too much data being uploaded without my consent.
  - scratchyone 2y ago
    If they sent a completely randomly generated integer from your phone without consent, would that be okay with you? Genuine question.
    - JTyQZSnP3cQGa8B 2y ago
      > If they sent [...] without consent, would that be okay with you? No. That would never be acceptable to me.
    - amiga386 2y ago

joe_the_user 2y ago

Just from memory when the scheme came up in earlier discussion. The system is essentially scanning for the signature for some known set of images of abuse so that it aims to capture abusers who would naively keep just these images on their machines. (It can't determine if a new image is abusive, notably). It's conceivable some number of (foolish and abusive) people will be caught this way and those favoring a long dragnet for this stuff will be happy. But this opens the possibility that a hacker could upload an image to an innocent person's computer and get that person arrested. Those favoring the long dragnet will naturally say the ends justify the means and you can't make an omelet without cracking a few eggs. Oh, "think of the children". Edit: Also worth adding that once a company is scanning user content to try to decide if the user is bad, it makes it that much easier to scan all kind of content in all kind of ways for all kinds of reasons. "for the good", naturally.

Thorrez 2y ago
>The system is essentially scanning for the signature for some known set of images of abuse Huh? The system is scanning for landmarks, not images of abuse. >people will be caught this way Due to the homomorphic encryption, I don't think Apple even knows whether the image matches a landmark in Apple's server database or not. So even if Apple put some images of abuse into its server database (which Apple claims only contains pictures of landmarks), I don't think Apple would know whether there was a match or not.
- zmgsabst 2y ago
  Does Apple explicitly say that? Or only that they don’t know which landmark it matched?
  - scratchyone 2y ago
    Fundamentally, vector search like this doesn't have a concept of something "matching" or "not matching". It's just a cosine similarity value. To determine if an image "matches", you have to check if that similarity is within some given threshold. If the results of the cosine similarity operation are encrypted (they are with HE), that wouldn't be possible to determine. The bigger privacy risk would be that the device routes the request to a specific database shard based on whichever has a center-point closest to the image embedding on the device. They take steps to protect this information such as third-party proxying to hide user IP addresses, as well as having devices send fake requests so that the server cannot tell which are real user data and which are fake data.

charles_f 2y ago

> please also write the parts that you disagree with. Thank you The problem invoked by the article is that data is being sent back to Apple by default. Saying "it's fine because it's encrypted" and "don't use apple if you're not fine with that" doesn't help. The post complains about a product that stored sensitive customer content locally now sends that data to Apple, and given the combination of abuse on privacy and horrendous, generalized security failures that we've seen across the industry, those concerns seem genuine. Your comment is very dismissive of these concerns, which would explain why it's being down voted.

dwaite 2y ago
Apple also makes a mandatory API call to captive.apple.com from every device with a web view, just about every time they connect to a network. If someone is willing to take a hardline stance that a vendor should inform the user and require opt-in consent for every type of exchange, they are going to have to run platforms specifically targeted toward that mindset.

zaroth 2y ago

Everything about what they did was absolutely fantastic and amazing, until they turned it on by default.

ustad 2y ago

You're extensively describing the technical implementation while missing the fundamental issue: Why is Apple enabling this feature by default for what is, essentially, a luxury photo search feature? Let's break this down: 1. Nobody is questioning whether Apple's technical implementation is sophisticated or secure. It clearly is. 2. Nobody is suggesting the privacy toggles don't work. They do. 3. The issue is about Apple deciding that automatically sending data about users' photos (regardless of how securely) is an acceptable default for a feature many users may never want or need. Consider the value proposition here: Apple invested significant engineering resources into complex homomorphic encryption and differential privacy... so users can search for landmarks in their photos? And they deemed this feature so essential that it should be enabled by default? This feels like using a golden vault with military-grade security to store grocery lists. Yes, the security is impressive, but that's not the point. The point is: Why are my grocery lists being moved to the vault without asking me first? A privacy-respecting approach would be: "Hey, would you like to enable enhanced landmark search in your photos? We've built some really impressive privacy protections to make this secure..." Instead of: "We've already started analyzing your photos for landmarks because we built really impressive privacy protections..." The sophistication of the technology doesn't justify making it the default for what is ultimately an optional convenience feature.

deleted 2y ago
[deleted]
slashdave 2y ago
> users may never want or need Are you assuming that Apple did not perform a market analysis when implementing this feature? I think that is unlikely, considering the effort.