3 ms·

It's not so much that the slippery slope didn't exist before — Apple has just greased it. > we aren't "closer" to that being a reality than we were before I w

It's not so much that the slippery slope didn't exist before — Apple has just greased it. > we aren't "closer" to that being a reality than we were before I want to believe this but it feels very much like we are. And I don't think it's purely psychological or alarmist. > it is and always has been one software update push away from being a reality As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away. It's still only “one software update push away” either way if you're counting releases, but it's not if you're counting tickets in a product backlog. It's easier for governments to expand an existing feature than to pressure companies to build one from scratch. Apple already built the feature security services dreamed of even though it had no legal obligation to (it has to report CSAM it finds; it does not have to actively search for it on your device). It's also easier to normalise the use of client-side scanning when one company has shipped it (without talking to the rest of the tech industry, and after declining invitations to talk to Cyber Policy/Internet Observatory teams at Stanford who are trying to help the industry as a whole with an improved collective approach). That pressure and the likely additional implementations of client-side scanning we'll see further expand the potential for abuse. I really enjoyed the Stanford discussion available on YouTube[1]: > @8:07 > …the other issue being that it wouldn't be terribly difficult to expand this client-side system to do all of the user's photos so we kind of have to trust the software to only apply this to things that are backed up to iCloud. > @17:24 > I think it's inevitable that some government — quite possibly the U.S. — is going to want to expand this to counter terrorism and, well, I think for a lot of people that sounds reasonable. The problem is that the definitions of terrorism are somewhat malleable… None of this feels alarmist to me. Yes, proprietary software is only ever an update away from throwing us all down the slope. But the slope is not a pure binary thing that is or isn't — companies can increase the gradient by decreasing the work it would take to hurt us, and that's what Apple has done here. [1]: https://www.youtube.com/watch?v=dbYZVNSOVy4 https://www.youtube.com/watch?v=dbYZVNSOVy4

helen___keller 5y ago
> As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away. I don't agree. Putting aside the "NeuralHash" algorithm that Apple made to try and match CSAM, scanning the files on a device is remarkably simple, and something Apple already does as part of your device's standard functioning (indexing files, attaching labels to pictures based on content detected by AI models, etc). Apple could have already implemented a secret image tag for "terrorist material" or "drug material" that is attached automatically to images, hidden from the user, and phoned home / reported to FBI when a threshold is met. How would you know this system doesn't already exist? Literally all the components for this system were already in place.
- modernerd 5y ago
  It would be great to have information about how much work this took. The only thing I've seen mentioning timescale was the leaked Apple/NCMEC memo: > Today marks the official public unveiling of Expanded Protections for Children, and I wanted to take a moment to thank each and every one of you for all of your hard work over the last few years. So this doesn't sound like a feature where all the puzzle pieces were in place and they just needed NeuralHash. Even if you casually “put aside” NeuralHash as you suggest, the amount of research and testing that has happened to ship the system they've described is not trivial. I stand by the idea that this was never a point release away. > Apple could have already implemented a secret image tag for terrorist material" or "drug material" that is attached automatically to images Apple cannot even consistently tag cats and dogs. There is no way it was ready to ship a feature that tags drug or terrorist material in a way that generates few enough false positives that agencies won't just turn it off. I do agree with you that we have no way to know what's running on closed-source devices (or even open-source ones, unless we personally audit the whole process from dust to device). For me, though, “you can't ever really know what's running on your device so why care about contentious new things you've just learned will definitely be running on your device” is not compelling. I might swallow 10 spiders a year in my sleep, but if someone offers to feed me one I think it's fair to decline.

helen___keller 5y ago

> As a software engineer it's hard to shake the feeling that we're now just a patch release away from misuse, where as last year we were a major update away. I don't agree. Putting aside the "NeuralHash" algorithm that Apple made to try and match CSAM, scanning the files on a device is remarkably simple, and something Apple already does as part of your device's standard functioning (indexing files, attaching labels to pictures based on content detected by AI models, etc). Apple could have already implemented a secret image tag for "terrorist material" or "drug material" that is attached automatically to images, hidden from the user, and phoned home / reported to FBI when a threshold is met. How would you know this system doesn't already exist? Literally all the components for this system were already in place.

modernerd 5y ago
It would be great to have information about how much work this took. The only thing I've seen mentioning timescale was the leaked Apple/NCMEC memo: > Today marks the official public unveiling of Expanded Protections for Children, and I wanted to take a moment to thank each and every one of you for all of your hard work over the last few years. So this doesn't sound like a feature where all the puzzle pieces were in place and they just needed NeuralHash. Even if you casually “put aside” NeuralHash as you suggest, the amount of research and testing that has happened to ship the system they've described is not trivial. I stand by the idea that this was never a point release away. > Apple could have already implemented a secret image tag for terrorist material" or "drug material" that is attached automatically to images Apple cannot even consistently tag cats and dogs. There is no way it was ready to ship a feature that tags drug or terrorist material in a way that generates few enough false positives that agencies won't just turn it off. I do agree with you that we have no way to know what's running on closed-source devices (or even open-source ones, unless we personally audit the whole process from dust to device). For me, though, “you can't ever really know what's running on your device so why care about contentious new things you've just learned will definitely be running on your device” is not compelling. I might swallow 10 spiders a year in my sleep, but if someone offers to feed me one I think it's fair to decline.