11 ms·

Read through it all, it still comes down to "trust us". Apple can sign and authorise an update at any time that will backdoor it, and the government is the str

by zmmmmm 2y ago

Read through it all, it still comes down to "trust us". Apple can sign and authorise an update at any time that will backdoor it, and the government is the stroke of a pen away from forcing them to, all completely silently. I get that there's benefit to what they are doing. But the problem of selling a message of trust is you absolutely have to be 100% truthful about it, and them failing to be transparent that people's data is still subject to access like this poisons the larger message they are selling.

buzzerbetrayed 2y ago
Your argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical. Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well: > Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good. > In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
- devjab 2y ago
  I think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part: > As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure. Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.
troad 2y ago
They already have root. Their software is closed source. There is absolutely nothing stopping them from uploading all of your data right now. If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.
- __MatrixMan__ 2y ago
  That's true, but also it should be possible to make an OS that people can trust without trusting you, and as users we should encourage movement in that direction.
  - abtinf 2y ago
    > should be possible What makes you think this?
    - brookst 2y ago
      It’s especially funny because I believe it is provably impossible. You’ll have to trust me that I’ve done the proof.
    - __MatrixMan__ 2y ago
      Because there are so freaking many of us, and some of us trust each other. If we were better at coordinating about which parts of the code we trust and to what degree, we could determine which parts of it are untrustworthy and patch the problem out of it. The GrapheneOS people are doing this, for example. It's not crazy to consider your device vendor as part of your threat model, because like it or not, they are a threat.
      - abtinf 2y ago
        That’s a good point. It would be interesting if there was a “git blame” style command, but that showed a trust score for every line/block based on who has touched it.

by zmmmmm 2y ago

buzzerbetrayed 2y ago

Your argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical. Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well: > Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good. > In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.

devjab 2y ago
I think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part: > As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure. Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.

troad 2y ago

They already have root. Their software is closed source. There is absolutely nothing stopping them from uploading all of your data right now. If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.

__MatrixMan__ 2y ago
That's true, but also it should be possible to make an OS that people can trust without trusting you, and as users we should encourage movement in that direction.
- abtinf 2y ago
  > should be possible What makes you think this?
  - brookst 2y ago
    It’s especially funny because I believe it is provably impossible. You’ll have to trust me that I’ve done the proof.
  - __MatrixMan__ 2y ago
    Because there are so freaking many of us, and some of us trust each other. If we were better at coordinating about which parts of the code we trust and to what degree, we could determine which parts of it are untrustworthy and patch the problem out of it. The GrapheneOS people are doing this, for example. It's not crazy to consider your device vendor as part of your threat model, because like it or not, they are a threat.
    - abtinf 2y ago
      That’s a good point. It would be interesting if there was a “git blame” style command, but that showed a trust score for every line/block based on who has touched it.