9 ms·

> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, et

by kemayo 5y ago

> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc. Although this is true, the same argument already applies to "your phone might be scanning all your photos and stealthily uploading them" -- Apple having announced this program doesn't seem to have changed the odds of that. At some point you have to trust your OS vendor.

majormajor 5y ago
If you're uploading to the cloud, you have to trust a lot more than just your OS vendor (well, in the default case, your OS vendor often == your cloud vendor, but the access is a lot greater once the data is on the cloud). And if your phone has the capability to upload to the cloud, then you have to trust your OS vendor to respect your wish if you disable it, etc. It's curious that this is the particular breaking point on the slope for people. The "on device" aspect just makes it more immediate feeling, I guess?
- kemayo 5y ago
  Yeah, it's weird. Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me. But this is clearly not a universal opinion. The most-optimistic take on this I can see is that this program could be the prelude to needing to trust less people. If Apple can turn on e2e encryption for photos, using this program as the PR shield from law enforcement to be able to do it, that'd leave us having to only trust the OS vendor.
  - notJim 5y ago
    > Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me. What I find interesting is that so many people find it worse to do it on device, because of the risk that they do it to photos you don't intend to upload. This is clearly where Apple got caught off-guard, because to them, on-device = private. It seems like the issue is really the mixing of on-device and off. People seem to be fine with on-device data that stays on-device, and relatively fine with the idea that Apple gets your content if you upload it to them. But when they analyze the data on-device, and then upload the results to the cloud, that really gets people.
    - shuckles 5y ago
      This seems like a necessary discussion to have in preparation for widespread, default end to end encryption.
    - rustymonday 5y ago
      Is this really surprising to you? I'm not trying to be rude, but this is an enormous distinction. In today's world, smartphones are basically an appendage of your body. They should not work to potentially incriminate its owner.
      - alwillis 5y ago
        They should not work to potentially incriminate its owner. But that ship has long sailed, right? Every packet that leaves a device potentially incriminates its owner. Every access point and router is a potential capture point.
        rustymonday 5y ago
        When I use a web service, I expect my data to be collected by the service, especially if it is free of charge. A device I own should not be allowed to collect and scan my data without my permission.
        alwillis 5y ago
        A device I own should not be allowed to collect and scan my data without my permission. It's not scanning; it's creating a cryptographic safety voucher for each photo you upload to iCloud Photos. And unless you reach a threshold of 30 CSAM images, Apple knows nothing about any of your photos.
        zekrioca 5y ago
        From the point of view of how image processing works, what is happening can indeed be called “scanning”.
- userbinator 5y ago
  Yes, you had to trust Apple, but the huge difference with this new thing is that hiding behind CSAM gives them far more (legally obligated, in fact --- because showing you the images those hashes came from would be illegal) plausible deniability and difficulty of verifying their claims. In other words, extracting the code and analysing it to determine that it does do what you expect is, although not easy, still legal. But the source, the CSAM itself, is illegal to possess, so you can't do that verification much less publish the results. It is this effective legal moat around those questioning the ultimate targets of this system which people are worried about.
  - majormajor 5y ago
    Surely they could do their image matching against all photos in iCloud without telling you in advance, and then you'd be in exactly the same boat? Google was doing this for email as early as 2014, for instance, with the same concerns about its extensibility raised by the ACLU: https://www.theguardian.com/technology/2014/aug/04/google-child-abuse-ncmec-internet-watch-gmail https://www.theguardian.com/technology/2014/aug/04/google-ch... So in a world where Apple pushes you to set up icloud photos by default, and can do whatever they want there, and other platforms have been doing this sort of of thing for years, it's a bit startling that "on device before you upload" vs "on uploaded content" triggers far more discontent? Maybe it's that Apple announced it at all, vs doing it relatively silently like the others? Apple has always had access to every photo on your device, after all.
    - pseudalopex 5y ago
      It isn't startling people trust they can opt out of iCloud photos.
      - kemayo 5y ago
        If you trust that you can opt out of iCloud Photos to avoid server-side scanning, trusting that this on-device scanning only happens as part of the iCloud Photos upload process (with the only way it submits the reports being as metadata attached to the photo-upload, as far as I can tell) seems equivalent. There's certainly a slippery-slope argument, where some future update might change that scanning behavior. But the system-as-currently-presented seems similarly trustable.
- himaraya 5y ago
  No, the threat model differs entirely. Local scanning introduces a whole host of single points of failure, including the 'independent auditor' & involuntary scans, that risk the privacy & security of all local files on a device. Cloud scanning largely precludes these potential vulnerabilities.
  - majormajor 5y ago
    Your phone threat model should already include "the OS author has full access to do whatever they want to whatever data is on my phone, and can change what they do any time they push out an update." I don't think anyone's necessarily being too upset or paranoid about THIS, but maybe everyone should also be a little less trusting of every closed OS - macOS, Windows, Android as provided by Google - that has root access too.
    - himaraya 5y ago
      Sure, but that doesn't change the fact that the vulnerabilities with local scanning remain a significant superset of cloud scanning's. Apple has built iOS off user trust & goodwill, unlike most other OSes.
      - shuckles 5y ago
        Cloud Scanning vulnerability: no transparency over data use. On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data. On the cloud, anything about your photos is fair game. Where does that fit in your set intersection?
        himaraya 5y ago
        > On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data. ...except you can't? Not sure where these assumptions come from.
        5y ago
almostdigital 5y ago
Yeah that's true, although to do some sort of mass scanning stealthily they would need a system exactly like what they built with this, if they tried to upload everything for scanning the data use would be enormous and give it away. I guess it comes down to that I don't trust an OS vendor that ships an A.I. based snitch program that they promise will be dormant.
- kemayo 5y ago
  Speaking cynically, I think that them having announced this program like they did makes it less likely that they have any sort of nefarious plans for it. There's a lot of attention being paid to it now, and it's on everyone's radar going forwards. If they actually wanted to be sneaky, we wouldn't have known about this for ages.
  - theonlybutlet 5y ago
    They'd have to be transparent about it as someone would easily figure it out.You have no way of verifying the contents of that hash database. once the infrastructure is in place (i.e. on your phone) it's a lot easier to expand on it. People have short memories and are easily desensitized, after a year or two of this, everyone will forget and we'll be in uproar about it expanding to include this or that...
  - smichel17 5y ago
    You're making the mistake of anthropomorphizing a corporation. Past a certain size, corporations start behaving less like people and more like computers, or maybe profit-maximizing sociopaths. The intent doesn't matter, because 5 or 10 years down the line, it'll likely be a totally different set of people making the decision. If you want to predict a corporation's behavior, you need to look at the constants (or at least, slower-changing things), like incentives, legal/technical limitations, and internal culture/structure of decision-making (e.g. How much agency do individual humans have?).
    - kemayo 5y ago
      I feel that I was stating the incentives, though. This being an area people are paying attention to makes it less likely they'll do unpopular things involving it, from a pure "we like good PR and profits" standpoint. They might sneak these things in elsewhere, but this specific on-device-scanning program has been shown to be a risk even at its current anodyne level.
slg 5y ago
Which is why I am confused by a lot of this backlash. Apple already controls the hardware, software, and services. I don't see why it really matters where in that chain the scanning is done when they control the entire system. If Apple can't be trusted with this control today, why did people trust them with this control a week ago?
- zwily 5y ago
  That’s where I’m at. They could have just started doing this without even saying anything at all.
  - jimworm 5y ago
    But in that case they would eventually be caught red-handed and won't get to do the "for the children" spiel and get it swept under the rug like it's about to be.
    - bredren 5y ago
      The goal is not for it to be swept under the rug. The goal is for it to deflect concerns over the coming Privacy Relay service.
      - threatofrain 5y ago
        Their private relay service appears orthogonal to CSAM… it won’t make criminals and child abusers easier or harder to catch, and it doesn’t affect how people use their iCloud Photos storage.
        bredren 5y ago
        These people are commonly prosecuted using evidence that includes server logs that showing their static IP address. Read evidence from past trials it is obvious. See also successful and failed attempts to subpoena this info from VPN services. Only people with iCloud will be using the relay. It is true on the surface the photos is disconnected from the use. However, Apple only needs a solid answer that handles the bad optics of what you can do with the Tor-like anonymity of iCloud Privacy Relay. However, if you look more closely, the CSAM service and its implementation are crafted exactly around the introduction of the relay.
g42gregory 5y ago
> At some point you have to trust your OS vendor. Yes, and we were trusting Apple. And now this trust is going away.
- rootusrootus 5y ago
  > now this trust is going away Is it really? There are some very loud voices making their discontent felt. But what does the Venn diagram look like between 'people who are loudly condemning Apple for this' and 'people who were vehemently anti-Apple to begin with'? My trust was shaken a bit, but the more I hear about the technology they've implemented, the more comfortable I am with it. And frankly, I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech.
  - matwood 5y ago
    > I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech. Yeah. I don't really understand the tech utopia feeling that Apple could simply turn on e2ee and ignore any future legislation to ban e2ee. The policy winds are clearly blowing towards limiting encryption in some fashion. Maybe this whole event will get people to pay more attention to policy...maybe.
Veserv 5y ago
Yes, they can technically already do so, but that is not the question. The question is what can they legally do and justify with high confidence in the event of a legal challenge. Changes to binding contractual terms that allow broad readings and provide legal justification for future overreach are dangerous. If they really are serious that they are going to use these new features in a highly limited way then they can put their money where their mouth is and add legally binding contractual terms that limit what they can do with serious consequences if they are found to be in breach. Non-binding marketing PR assurances that they will not abuse their contractually justified powers are no substitute for the iron fist of legal penalty clause.
m4rtink 5y ago
What about trust-but-verify ? If the OS was open source and supported reproducible builds, you would not have to trust them, you could verify what it actually does & make sure the signed binaries they ship you actually correspond to the source code. Once kinda wonders what they want to hide if they talks so much about user privacy yet don't provide any means for users to verify their claims.
ianmiers 5y ago
What happens if someone tries to coerce Apple into writing backdoor code? Engineers at Apple could resist, resign, slow roll the design and engineering process. They could leak it and it would get killed. Things would have to get very very bad for that kind of pressure to work. On the other hand, once Apple has written a backdoor enthusiastically themselves, it's a lot easier to force someone to change how it can be used. The changes are small and compliance can be immediately verified and refusal punished. To take it to its logical extreme: you cannot really fire or execute people who delay something (especially if you lake the expertise to tell how long it should take). But you can fire or execute people who refuse to flip a switch. This technology deeply erodes Apple and its engineers' ability to resist future pressure. And the important bit here is there adversary isn't all powerful. It can coerce you to do things in secret, but its power isn't unlimited. See what happened with yahoo.[0] https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclusive-yahoo-secretly-scanned-customer-emails-for-u-s-intelligence-sources-idUSKCN1241YT https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclu...

by kemayo 5y ago

majormajor 5y ago

If you're uploading to the cloud, you have to trust a lot more than just your OS vendor (well, in the default case, your OS vendor often == your cloud vendor, but the access is a lot greater once the data is on the cloud). And if your phone has the capability to upload to the cloud, then you have to trust your OS vendor to respect your wish if you disable it, etc. It's curious that this is the particular breaking point on the slope for people. The "on device" aspect just makes it more immediate feeling, I guess?

kemayo 5y ago
Yeah, it's weird. Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me. But this is clearly not a universal opinion. The most-optimistic take on this I can see is that this program could be the prelude to needing to trust less people. If Apple can turn on e2e encryption for photos, using this program as the PR shield from law enforcement to be able to do it, that'd leave us having to only trust the OS vendor.
- notJim 5y ago
  > Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me. What I find interesting is that so many people find it worse to do it on device, because of the risk that they do it to photos you don't intend to upload. This is clearly where Apple got caught off-guard, because to them, on-device = private. It seems like the issue is really the mixing of on-device and off. People seem to be fine with on-device data that stays on-device, and relatively fine with the idea that Apple gets your content if you upload it to them. But when they analyze the data on-device, and then upload the results to the cloud, that really gets people.
  - shuckles 5y ago
    This seems like a necessary discussion to have in preparation for widespread, default end to end encryption.
  - rustymonday 5y ago
    Is this really surprising to you? I'm not trying to be rude, but this is an enormous distinction. In today's world, smartphones are basically an appendage of your body. They should not work to potentially incriminate its owner.
    - alwillis 5y ago
      They should not work to potentially incriminate its owner. But that ship has long sailed, right? Every packet that leaves a device potentially incriminates its owner. Every access point and router is a potential capture point.
      - rustymonday 5y ago
        When I use a web service, I expect my data to be collected by the service, especially if it is free of charge. A device I own should not be allowed to collect and scan my data without my permission.
        alwillis 5y ago
        A device I own should not be allowed to collect and scan my data without my permission. It's not scanning; it's creating a cryptographic safety voucher for each photo you upload to iCloud Photos. And unless you reach a threshold of 30 CSAM images, Apple knows nothing about any of your photos.
        zekrioca 5y ago
        From the point of view of how image processing works, what is happening can indeed be called “scanning”.
userbinator 5y ago
Yes, you had to trust Apple, but the huge difference with this new thing is that hiding behind CSAM gives them far more (legally obligated, in fact --- because showing you the images those hashes came from would be illegal) plausible deniability and difficulty of verifying their claims. In other words, extracting the code and analysing it to determine that it does do what you expect is, although not easy, still legal. But the source, the CSAM itself, is illegal to possess, so you can't do that verification much less publish the results. It is this effective legal moat around those questioning the ultimate targets of this system which people are worried about.
- majormajor 5y ago
  Surely they could do their image matching against all photos in iCloud without telling you in advance, and then you'd be in exactly the same boat? Google was doing this for email as early as 2014, for instance, with the same concerns about its extensibility raised by the ACLU: https://www.theguardian.com/technology/2014/aug/04/google-child-abuse-ncmec-internet-watch-gmail https://www.theguardian.com/technology/2014/aug/04/google-ch... So in a world where Apple pushes you to set up icloud photos by default, and can do whatever they want there, and other platforms have been doing this sort of of thing for years, it's a bit startling that "on device before you upload" vs "on uploaded content" triggers far more discontent? Maybe it's that Apple announced it at all, vs doing it relatively silently like the others? Apple has always had access to every photo on your device, after all.
  - pseudalopex 5y ago
    It isn't startling people trust they can opt out of iCloud photos.
    - kemayo 5y ago
      If you trust that you can opt out of iCloud Photos to avoid server-side scanning, trusting that this on-device scanning only happens as part of the iCloud Photos upload process (with the only way it submits the reports being as metadata attached to the photo-upload, as far as I can tell) seems equivalent. There's certainly a slippery-slope argument, where some future update might change that scanning behavior. But the system-as-currently-presented seems similarly trustable.
himaraya 5y ago
No, the threat model differs entirely. Local scanning introduces a whole host of single points of failure, including the 'independent auditor' & involuntary scans, that risk the privacy & security of all local files on a device. Cloud scanning largely precludes these potential vulnerabilities.
- majormajor 5y ago
  Your phone threat model should already include "the OS author has full access to do whatever they want to whatever data is on my phone, and can change what they do any time they push out an update." I don't think anyone's necessarily being too upset or paranoid about THIS, but maybe everyone should also be a little less trusting of every closed OS - macOS, Windows, Android as provided by Google - that has root access too.
  - himaraya 5y ago
    Sure, but that doesn't change the fact that the vulnerabilities with local scanning remain a significant superset of cloud scanning's. Apple has built iOS off user trust & goodwill, unlike most other OSes.
    - shuckles 5y ago
      Cloud Scanning vulnerability: no transparency over data use. On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data. On the cloud, anything about your photos is fair game. Where does that fit in your set intersection?
      - himaraya 5y ago
        > On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data. ...except you can't? Not sure where these assumptions come from.
        5y ago

almostdigital 5y ago

Yeah that's true, although to do some sort of mass scanning stealthily they would need a system exactly like what they built with this, if they tried to upload everything for scanning the data use would be enormous and give it away. I guess it comes down to that I don't trust an OS vendor that ships an A.I. based snitch program that they promise will be dormant.

kemayo 5y ago
Speaking cynically, I think that them having announced this program like they did makes it less likely that they have any sort of nefarious plans for it. There's a lot of attention being paid to it now, and it's on everyone's radar going forwards. If they actually wanted to be sneaky, we wouldn't have known about this for ages.
- theonlybutlet 5y ago
  They'd have to be transparent about it as someone would easily figure it out.You have no way of verifying the contents of that hash database. once the infrastructure is in place (i.e. on your phone) it's a lot easier to expand on it. People have short memories and are easily desensitized, after a year or two of this, everyone will forget and we'll be in uproar about it expanding to include this or that...
- smichel17 5y ago
  You're making the mistake of anthropomorphizing a corporation. Past a certain size, corporations start behaving less like people and more like computers, or maybe profit-maximizing sociopaths. The intent doesn't matter, because 5 or 10 years down the line, it'll likely be a totally different set of people making the decision. If you want to predict a corporation's behavior, you need to look at the constants (or at least, slower-changing things), like incentives, legal/technical limitations, and internal culture/structure of decision-making (e.g. How much agency do individual humans have?).
  - kemayo 5y ago
    I feel that I was stating the incentives, though. This being an area people are paying attention to makes it less likely they'll do unpopular things involving it, from a pure "we like good PR and profits" standpoint. They might sneak these things in elsewhere, but this specific on-device-scanning program has been shown to be a risk even at its current anodyne level.

slg 5y ago

Which is why I am confused by a lot of this backlash. Apple already controls the hardware, software, and services. I don't see why it really matters where in that chain the scanning is done when they control the entire system. If Apple can't be trusted with this control today, why did people trust them with this control a week ago?

zwily 5y ago
That’s where I’m at. They could have just started doing this without even saying anything at all.
- jimworm 5y ago
  But in that case they would eventually be caught red-handed and won't get to do the "for the children" spiel and get it swept under the rug like it's about to be.
  - bredren 5y ago
    The goal is not for it to be swept under the rug. The goal is for it to deflect concerns over the coming Privacy Relay service.
    - threatofrain 5y ago
      Their private relay service appears orthogonal to CSAM… it won’t make criminals and child abusers easier or harder to catch, and it doesn’t affect how people use their iCloud Photos storage.
      - bredren 5y ago
        These people are commonly prosecuted using evidence that includes server logs that showing their static IP address. Read evidence from past trials it is obvious. See also successful and failed attempts to subpoena this info from VPN services. Only people with iCloud will be using the relay. It is true on the surface the photos is disconnected from the use. However, Apple only needs a solid answer that handles the bad optics of what you can do with the Tor-like anonymity of iCloud Privacy Relay. However, if you look more closely, the CSAM service and its implementation are crafted exactly around the introduction of the relay.

g42gregory 5y ago

> At some point you have to trust your OS vendor. Yes, and we were trusting Apple. And now this trust is going away.

rootusrootus 5y ago
> now this trust is going away Is it really? There are some very loud voices making their discontent felt. But what does the Venn diagram look like between 'people who are loudly condemning Apple for this' and 'people who were vehemently anti-Apple to begin with'? My trust was shaken a bit, but the more I hear about the technology they've implemented, the more comfortable I am with it. And frankly, I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech.
- matwood 5y ago
  > I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech. Yeah. I don't really understand the tech utopia feeling that Apple could simply turn on e2ee and ignore any future legislation to ban e2ee. The policy winds are clearly blowing towards limiting encryption in some fashion. Maybe this whole event will get people to pay more attention to policy...maybe.

Veserv 5y ago

Yes, they can technically already do so, but that is not the question. The question is what can they legally do and justify with high confidence in the event of a legal challenge. Changes to binding contractual terms that allow broad readings and provide legal justification for future overreach are dangerous. If they really are serious that they are going to use these new features in a highly limited way then they can put their money where their mouth is and add legally binding contractual terms that limit what they can do with serious consequences if they are found to be in breach. Non-binding marketing PR assurances that they will not abuse their contractually justified powers are no substitute for the iron fist of legal penalty clause.

m4rtink 5y ago

What about trust-but-verify ? If the OS was open source and supported reproducible builds, you would not have to trust them, you could verify what it actually does & make sure the signed binaries they ship you actually correspond to the source code. Once kinda wonders what they want to hide if they talks so much about user privacy yet don't provide any means for users to verify their claims.

ianmiers 5y ago

What happens if someone tries to coerce Apple into writing backdoor code? Engineers at Apple could resist, resign, slow roll the design and engineering process. They could leak it and it would get killed. Things would have to get very very bad for that kind of pressure to work. On the other hand, once Apple has written a backdoor enthusiastically themselves, it's a lot easier to force someone to change how it can be used. The changes are small and compliance can be immediately verified and refusal punished. To take it to its logical extreme: you cannot really fire or execute people who delay something (especially if you lake the expertise to tell how long it should take). But you can fire or execute people who refuse to flip a switch. This technology deeply erodes Apple and its engineers' ability to resist future pressure. And the important bit here is there adversary isn't all powerful. It can coerce you to do things in secret, but its power isn't unlimited. See what happened with yahoo.[0] https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclusive-yahoo-secretly-scanned-customer-emails-for-u-s-intelligence-sources-idUSKCN1241YT https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclu...