6 ms·

I take issue with most of the alarmism about this CSAM scanning. Not because I think our devices scanning our content is okay, but because of the implication th

I take issue with most of the alarmism about this CSAM scanning. Not because I think our devices scanning our content is okay, but because of the implication that there's now a slippery slope that didn't exist before. For example, from the article: > While today it has been purpose-built for CSAM, and it can be deactivated simply by shutting off iCloud Photo Library syncing, it still feels like a line has been crossed Two simple facts: (1) The system, as described today, isn't any more invasive then the existing CSAM scanning technologies that exist on all major cloud storage systems (arguably it's less invasive - no external systems look at your photos unless your phone flags enough of your photos as CSAM, which brings it to a manual review stage) (2) Auto-updating root-level proprietary software can be updated to any level of invasion of privacy at any time for any reason the provider wishes. We aren't any closer to full-invasion-of-privacy with iPhone than we were before; it is and always has been one single update away. In fact, we don't know if it's already there on iPhone or any other proprietary system such as Windows, Chromebook, etc. Who knows what backdoors exist on these systems? If you truly believe that you need full control and a system you fully trust, don't get a device that runs proprietary software. If you're okay with a device that isn't fully trustworthy, but appears to be benevolent, then iPhone isn't any worse than it was a month ago. Until there's evidence otherwise, iPhone will continue to be as trustworthy as any other proprietary closed-source system. If you need more than that, please contribute to projects that aim to produce a modern, functional FOSS smartphone.

stetrain 5y ago
I agree with (2). But I do think there is a fundamental (human, psychological, emotional) difference between a tool running on a server and one running on my device. The mental and ethical lines are clear when I hand content to a server, that's a contract I can understand and agree to. When it runs on my device, even if it's because of the option to send it to the server, that feels fuzzier. As Ben Thompson put it it's the difference between Capability and Policy. This is now rubbing in our faces that the privacy is only in the policy, not the capabilities of the software. And the capabilities of propriety software are pretty knowable, through a combination of inspection and long term observation. The end result isn't any different today, but it highlights the political position that Apple and others are being put in. I honestly don't fault Apple for this on anything except their clearly poor messaging, but they are getting pulled from all sides by governments who don't like the idea of them actually implementing a fully closed and end-to-end encrypted system for user data.
- helen___keller 5y ago
  > But I do think there is a fundamental (human, psychological, emotional) difference between a tool running on a server and one running on my device. I agree, but it's frustrating to me because it's an illusion: if your device is running proprietary software, you don't control the device. At that point there's little difference between your device and a server in the cloud, aside from technical characteristics like latency and compute power.
  - bcrosby95 5y ago
    The difference is I supposedly own my device. This is a line we use for all sorts of things, such as, say, hacking charges.
    - helen___keller 5y ago
      You explicitly do not own the software running on the device. It's Apple's software running both on your device and on their servers.
  - stetrain 5y ago
    I think Apple is getting so much heat for this because they have at least maintained an outward appearance of separation between device and cloud for so long. Some of this was just due to being further behind on the SaaS ramp than MS and Google. Everyone expects a Google-services enabled Android device to basically be a cloud thin client, but less so an iPhone. And Apple has actually stood up for encryption and not putting in openly exploitable backdoor capabilities just to appease law enforcement, see their back and forth with the FBI a couple of years ago. Apple seems to be coming up with awkward solutions as they try to do the "right thing" in some places (more end to end encryption) while governments / law enforcement don't want hidden data because of "the children", terrorism, etc. They announced this proudly as though they had finally found the holy grail to negotiate this quagmire but the messaging did not go over at all how they seemed to expect. I don't think there's a path forward for fully open and non-proprietary software that everyone uses. It hasn't happened on the desktop so I doubt it will on mobile devices. And most people want their devices to be cloud terminals, not sovereign system states. So I think it comes back down to a legal and political issue of what requirements governments put on these tech companies, how data and privacy protections are guaranteed (or forced to be broken), and how that gets enforced.
tadfisher 5y ago
I agree with you, but you're painting the alarmists with a broad stroke. There's room for those who trust Apple with their data, are fine with the implications of server-side data storage, and are not fine with their devices performing the scan on offline data.
- bengale 5y ago
  It only scans data that’s going online though.
- helen___keller 5y ago
  > There's room for those who trust Apple with their data, are fine with the implications of server-side data storage, and are not fine with their devices performing the scan on offline data. Right, my point is that the latter (scanning of offline data) does not yet have any evidence of occurring, and we aren't "closer" to that being a reality than we were before: it is and always has been one software update push away from being a reality.
  - modernerd 5y ago
    It's not so much that the slippery slope didn't exist before — Apple has just greased it. > we aren't "closer" to that being a reality than we were before I want to believe this but it feels very much like we are. And I don't think it's purely psychological or alarmist. > it is and always has been one software update push away from being a reality As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away. It's still only “one software update push away” either way if you're counting releases, but it's not if you're counting tickets in a product backlog. It's easier for governments to expand an existing feature than to pressure companies to build one from scratch. Apple already built the feature security services dreamed of even though it had no legal obligation to (it has to report CSAM it finds; it does not have to actively search for it on your device). It's also easier to normalise the use of client-side scanning when one company has shipped it (without talking to the rest of the tech industry, and after declining invitations to talk to Cyber Policy/Internet Observatory teams at Stanford who are trying to help the industry as a whole with an improved collective approach). That pressure and the likely additional implementations of client-side scanning we'll see further expand the potential for abuse. I really enjoyed the Stanford discussion available on YouTube[1]: > @8:07 > …the other issue being that it wouldn't be terribly difficult to expand this client-side system to do all of the user's photos so we kind of have to trust the software to only apply this to things that are backed up to iCloud. > @17:24 > I think it's inevitable that some government — quite possibly the U.S. — is going to want to expand this to counter terrorism and, well, I think for a lot of people that sounds reasonable. The problem is that the definitions of terrorism are somewhat malleable… None of this feels alarmist to me. Yes, proprietary software is only ever an update away from throwing us all down the slope. But the slope is not a pure binary thing that is or isn't — companies can increase the gradient by decreasing the work it would take to hurt us, and that's what Apple has done here. [1]: https://www.youtube.com/watch?v=dbYZVNSOVy4 https://www.youtube.com/watch?v=dbYZVNSOVy4
deleted 5y ago
[deleted]
the8472 5y ago
> that exist on all major cloud storage systems Except for those cloud storage systems that have offered client-side encryption from day 1. Pretending that server-side scanning is something naturally given and should be accepted as baseline biases the arguments that follow.
rickspencer3 5y ago
I'm also surprised that this is the thing that gets people up in arms about iPhones. I mean, you can't even write legitimate application for yourself and run it on your own phone without insane hoops to jump through. Not to mention that you can't buy an iPhone and decide to change the OS or whatever with the hardware, even if Apple stops supporting it. I have no issue with people who give of very very basic user freedom because they want to opt into the iOS "ecosystem", but I don't see how this particular thing would push anyone of the ledge.
- DavideNL 5y ago
  Simple, the line that was crossed is scanning your local files versus scanning cloud files (even though they promise this system is disabled if you disabled icloud photos.) Once this system is in place, it takes only 1 tiny adjustment/exploit to scan other private stuff on your devices, which you would never upload to a cloud.
  - helen___keller 5y ago
    > it takes only 1 tiny adjustment/exploit to scan other private stuff on your devices This has always been true, there's no "Once this system is in place" qualifier necessary. This is the reality of running someone else's proprietary code at the root level on your device.
    - DavideNL 5y ago
      I respectfully disagree :) that's black/white thinking, by that reasoning all software not inspected by yourself, and having some automated update system, can be considered unsafe. The world is not just black and white...
  - nomel 5y ago
    > scanning your local files versus scanning cloud files Correction, scanning cloud destined files, as a step of the upload pipeline. Local files not destined for the cloud are not scanned. It's a scan happening before the network stack of the pipeline, rather than after. But it still is already in the pipeline, an its way, at the point of scan (at least that's my understanding of their docs).
noelsusman 5y ago
Agreed, and I'll add that the existing cloud systems have been running for a long time and have yet to slip into the various dystopian hypotheticals I've seen tossed around. Google isn't scanning Drive for documents containing anti-government sentiments. I'd be very interested in evidence that suggests otherwise.
deleted 5y ago
[deleted]
stjohnswarts 5y ago
Go ahead and take issue. You missed the part about them turning your own device against you and normalizing it as "oh it's just a little thing". This is huge and you shouldn't downplay it. Phones are at the center of our digital life and this is allowing a government proxy in to (first steps, likely to be followed by many more). I am quite glad that people are "overreacting" to it. It seems like you want us to just eat our grass like sheep. I will keep raising issues with it not matter how many times people say "alarmist" or "must be a pedo" or whatever is the insult of the day.
lamontcg 5y ago
It isn't necessary to believe in a slippery slope to be very alarmed at this. Their 1-in-a-trillion estimate of false positives is very likely total horseshit (there was a recent article on dissecting that claim from someone familiar with CSAM). It also opens up a "swatting" avenue where if someone hacks your phone they can upload CSAM images and apple will sicthe authorities on you. Good luck with that. Really we need to bring back comp.risks and start training ourselves to think a bit more cynically about the downsides of technology.
cyanydeez 5y ago
csam will surely be followed by copyright scams

stetrain 5y ago

I agree with (2). But I do think there is a fundamental (human, psychological, emotional) difference between a tool running on a server and one running on my device. The mental and ethical lines are clear when I hand content to a server, that's a contract I can understand and agree to. When it runs on my device, even if it's because of the option to send it to the server, that feels fuzzier. As Ben Thompson put it it's the difference between Capability and Policy. This is now rubbing in our faces that the privacy is only in the policy, not the capabilities of the software. And the capabilities of propriety software are pretty knowable, through a combination of inspection and long term observation. The end result isn't any different today, but it highlights the political position that Apple and others are being put in. I honestly don't fault Apple for this on anything except their clearly poor messaging, but they are getting pulled from all sides by governments who don't like the idea of them actually implementing a fully closed and end-to-end encrypted system for user data.

helen___keller 5y ago
> But I do think there is a fundamental (human, psychological, emotional) difference between a tool running on a server and one running on my device. I agree, but it's frustrating to me because it's an illusion: if your device is running proprietary software, you don't control the device. At that point there's little difference between your device and a server in the cloud, aside from technical characteristics like latency and compute power.
- bcrosby95 5y ago
  The difference is I supposedly own my device. This is a line we use for all sorts of things, such as, say, hacking charges.
  - helen___keller 5y ago
    You explicitly do not own the software running on the device. It's Apple's software running both on your device and on their servers.
- stetrain 5y ago
  I think Apple is getting so much heat for this because they have at least maintained an outward appearance of separation between device and cloud for so long. Some of this was just due to being further behind on the SaaS ramp than MS and Google. Everyone expects a Google-services enabled Android device to basically be a cloud thin client, but less so an iPhone. And Apple has actually stood up for encryption and not putting in openly exploitable backdoor capabilities just to appease law enforcement, see their back and forth with the FBI a couple of years ago. Apple seems to be coming up with awkward solutions as they try to do the "right thing" in some places (more end to end encryption) while governments / law enforcement don't want hidden data because of "the children", terrorism, etc. They announced this proudly as though they had finally found the holy grail to negotiate this quagmire but the messaging did not go over at all how they seemed to expect. I don't think there's a path forward for fully open and non-proprietary software that everyone uses. It hasn't happened on the desktop so I doubt it will on mobile devices. And most people want their devices to be cloud terminals, not sovereign system states. So I think it comes back down to a legal and political issue of what requirements governments put on these tech companies, how data and privacy protections are guaranteed (or forced to be broken), and how that gets enforced.

tadfisher 5y ago

I agree with you, but you're painting the alarmists with a broad stroke. There's room for those who trust Apple with their data, are fine with the implications of server-side data storage, and are not fine with their devices performing the scan on offline data.

bengale 5y ago
It only scans data that’s going online though.
helen___keller 5y ago
> There's room for those who trust Apple with their data, are fine with the implications of server-side data storage, and are not fine with their devices performing the scan on offline data. Right, my point is that the latter (scanning of offline data) does not yet have any evidence of occurring, and we aren't "closer" to that being a reality than we were before: it is and always has been one software update push away from being a reality.
- modernerd 5y ago
  It's not so much that the slippery slope didn't exist before — Apple has just greased it. > we aren't "closer" to that being a reality than we were before I want to believe this but it feels very much like we are. And I don't think it's purely psychological or alarmist. > it is and always has been one software update push away from being a reality As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away. It's still only “one software update push away” either way if you're counting releases, but it's not if you're counting tickets in a product backlog. It's easier for governments to expand an existing feature than to pressure companies to build one from scratch. Apple already built the feature security services dreamed of even though it had no legal obligation to (it has to report CSAM it finds; it does not have to actively search for it on your device). It's also easier to normalise the use of client-side scanning when one company has shipped it (without talking to the rest of the tech industry, and after declining invitations to talk to Cyber Policy/Internet Observatory teams at Stanford who are trying to help the industry as a whole with an improved collective approach). That pressure and the likely additional implementations of client-side scanning we'll see further expand the potential for abuse. I really enjoyed the Stanford discussion available on YouTube[1]: > @8:07 > …the other issue being that it wouldn't be terribly difficult to expand this client-side system to do all of the user's photos so we kind of have to trust the software to only apply this to things that are backed up to iCloud. > @17:24 > I think it's inevitable that some government — quite possibly the U.S. — is going to want to expand this to counter terrorism and, well, I think for a lot of people that sounds reasonable. The problem is that the definitions of terrorism are somewhat malleable… None of this feels alarmist to me. Yes, proprietary software is only ever an update away from throwing us all down the slope. But the slope is not a pure binary thing that is or isn't — companies can increase the gradient by decreasing the work it would take to hurt us, and that's what Apple has done here. [1]: https://www.youtube.com/watch?v=dbYZVNSOVy4 https://www.youtube.com/watch?v=dbYZVNSOVy4

deleted 5y ago

[deleted]

the8472 5y ago

> that exist on all major cloud storage systems Except for those cloud storage systems that have offered client-side encryption from day 1. Pretending that server-side scanning is something naturally given and should be accepted as baseline biases the arguments that follow.

rickspencer3 5y ago

I'm also surprised that this is the thing that gets people up in arms about iPhones. I mean, you can't even write legitimate application for yourself and run it on your own phone without insane hoops to jump through. Not to mention that you can't buy an iPhone and decide to change the OS or whatever with the hardware, even if Apple stops supporting it. I have no issue with people who give of very very basic user freedom because they want to opt into the iOS "ecosystem", but I don't see how this particular thing would push anyone of the ledge.

DavideNL 5y ago
Simple, the line that was crossed is scanning your local files versus scanning cloud files (even though they promise this system is disabled if you disabled icloud photos.) Once this system is in place, it takes only 1 tiny adjustment/exploit to scan other private stuff on your devices, which you would never upload to a cloud.
- helen___keller 5y ago
  > it takes only 1 tiny adjustment/exploit to scan other private stuff on your devices This has always been true, there's no "Once this system is in place" qualifier necessary. This is the reality of running someone else's proprietary code at the root level on your device.
  - DavideNL 5y ago
    I respectfully disagree :) that's black/white thinking, by that reasoning all software not inspected by yourself, and having some automated update system, can be considered unsafe. The world is not just black and white...
- nomel 5y ago
  > scanning your local files versus scanning cloud files Correction, scanning cloud destined files, as a step of the upload pipeline. Local files not destined for the cloud are not scanned. It's a scan happening before the network stack of the pipeline, rather than after. But it still is already in the pipeline, an its way, at the point of scan (at least that's my understanding of their docs).

noelsusman 5y ago

Agreed, and I'll add that the existing cloud systems have been running for a long time and have yet to slip into the various dystopian hypotheticals I've seen tossed around. Google isn't scanning Drive for documents containing anti-government sentiments. I'd be very interested in evidence that suggests otherwise.

deleted 5y ago

[deleted]

stjohnswarts 5y ago

Go ahead and take issue. You missed the part about them turning your own device against you and normalizing it as "oh it's just a little thing". This is huge and you shouldn't downplay it. Phones are at the center of our digital life and this is allowing a government proxy in to (first steps, likely to be followed by many more). I am quite glad that people are "overreacting" to it. It seems like you want us to just eat our grass like sheep. I will keep raising issues with it not matter how many times people say "alarmist" or "must be a pedo" or whatever is the insult of the day.

lamontcg 5y ago

It isn't necessary to believe in a slippery slope to be very alarmed at this. Their 1-in-a-trillion estimate of false positives is very likely total horseshit (there was a recent article on dissecting that claim from someone familiar with CSAM). It also opens up a "swatting" avenue where if someone hacks your phone they can upload CSAM images and apple will sicthe authorities on you. Good luck with that. Really we need to bring back comp.risks and start training ourselves to think a bit more cynically about the downsides of technology.

cyanydeez 5y ago