13 ms·

The thing with cloud and with anything related to it, anything that connects to the internet somehow... is that, unless it's open source and the servers decentr

The thing with cloud and with anything related to it, anything that connects to the internet somehow... is that, unless it's open source and the servers decentralized, you are always trusting SOMEONE. Sure, Apple might make their best to ensure nobody – but them – have access to your data... but Apple controls all the end points. It controls the updates your iPhone receives, it controls the servers where this happens. Like, they are so many opportunities for them to find what you are doing. It reminds me of this article "Web-based cryptography is always snake oil" https://www.devever.net/~hl/webcrypto https://www.devever.net/~hl/webcrypto And to be fair, this doesn't apply only to this case. Even the data you have stored locally, Apple could access it if they wanted, they sure have power to do it if they so wish or were ordered by the government. They might have done it already and just didn't told anyone for obvious reasons. So, I would argue the best you could say is that it's private in the sense that only Apples knows/can know what you are doing rather than a larger number of entities . Which, you could argue it's a win when the alternatives will leak your data to many more parts... But still far away from being this unbreakable cryptography that it's portrayed it to be.

seydor 2y ago
> if wanted. Or if someone compels them to
noahtallen 2y ago
I don’t think that’s completely fair. It basically puts Apple in the same bucket as Google or OpenAI. Google obviously tracks everything you do for ads, recommendations, AI, you name it. They don’t even hide it, it’s a core part of their business model. Apple, on the other hand, has made a pretty serious effort to ensure that no employee can access your data on these AI systems. That’s hugely different! They’re going as far as to severely restrict logging and observability and even building and designing their own chips and operating systems. And ensuring that clients will refuse to talk to non-audited systems. Yes, we can’t take Apple’s word for it. But I think the third party audits are a huge part of how we trust, and also verify, that this system will be private. I don’t think it’s far to claim that “Apple knows what you’re doing.” That implies that some one, at some level at Apple can at some point access the data sent from your device to this private cloud. That does not seem to be true. I think another facet of trust here is that a rather big part of Apple’s business model is privacy. They’ve been very successful financially by creating products that generate money in other ways, and it’s very much not necessary or even a sound business idea for them to do something else. While I think it’s fair to be skeptical about the claims without 3rd party verification, I don’t think it’s fair to say that Apple’s approach isn’t better for your data and privacy than openAI or Google. (Which I think is the broad implication — openAI tracks prompts for its own model training, not to resell, so it’s also “only openAI knows what your doing.”)
- verisimi 2y ago
  It's completely fair, because regardless of third party audits, chips, etc, there are backdoors right along the line, that are going to provide Apple and the government with secret legal access to your data. They can simply go to a secret court, receive a secret judgment, and be authorised to secretly view your data. Does anyone really think this is not already the case? There is no transparency. A licensed third party auditor would not be able to tell you this. We have to operate with the awareness that all data online is already not private - no need to pretend/imagine that Apple's marketing is actually true, and that it is possible to buy online privacy utopia.
  - theshrike79 2y ago
    The best protection against "secret orders" is to use mathematics. Build your system so that it can't be decrypted, don't log anything etc. Mullvad has been doing this with VPNs and law enforcement has tested it - there's nothing for them to get. Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone. There's no reason why they wouldn't go to same lengths on their private cloud compute. It's the one thing they can do that Google can't.
    - verisimi 2y ago
      > Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone. Right, but I have no reason to think that this isn't a marketing ploy either, just another story. There is simply no way that Apple is as big as it is, without providing whatever data the government requires. Corporations and governments are not your friend.
      - theshrike79 2y ago
        Apple will obey government orders to give data they have and can access. No government order short of targeting a specific backdoored update to a specific person will allow them to give data they can't access. And if you're doing something that can make a TLA force Apple to create a targeted iOS update just for you, it's not something regular people can or should worry about. Apple keeps normal people safe from mass surveillance, being protected from CIA/NSA required going Full Snowden and it's not a technological problem, you need to change the way you live.
TeMPOraL 2y ago
> unless it's open source and the servers decentralized, you are always trusting SOMEONE Specifically, open-source and self-hostable. Open source doesn't save you if people can't run their own servers, because you never know whether what's in the public repo is the exact same thing that's running on the cloud servers.
- nardi 2y ago
  This is what the “attestation” bit is supposed to take care of—if it works, which I’m assuming it will, because they’re open sourcing it for security auditing.
- jjav 2y ago
  > exact same thing that's running on the cloud servers What runs on the servers isn't actually very important. Why? Becuase even if you could somehow know with 100% certainty that what a server runs is the same code you can see, any provider is still subject to all kinds of court orders. What matters is the client code. If you can audit the client code (or better yet, build your own compatible client based on API specs) then you know for sure what the server side sees. If everything is encrypted locally with keys only you control, it doesn't matter what runs on the server.
  - flakeoil 2y ago
    But in this use case of AI in the cloud I suppose it's not possible to send encrypted data which only you have the keys to as that makes the data useless and thus no AI processing in the cloud can be made. So the whole point of AI in the cloud vs. AI on device goes away.
- dwaite 2y ago
  You can by having an attestation of the signed software components up from the secure boot process, and having the client device validate said attestation corresponds to the known public version of each component, and randomize client connections across infrastructure. Other than obvious "open source software isn't perfectly secure" attack scenarios, this would require a non-targeted hardware attack, where the entire infrastructure would need to misinterpret the software or misrepresent the chain of custody. I believe this is one of the protections Apple is attempting to implement here.
detourdog 2y ago
If one has to use tech one has to trust someone. Apple has focused on the individual using computers since inception. They have maintained a consistent message and have a good track record. I will trust them because the alternatives I see are scattered and unfocused.
nl 2y ago
This isn't right. If you trust math you can prove the software is what they say it is. Yes it is work to do this, but this is a big step forward.
- ADeerAppeared 2y ago
  The only thing the math tells you is that the server software gave you a correct key. It does not tell you how it got that key. A compromised server would send you the key all the same. You still have to trust in the security infrastructure. Trust that Apple is running the hardware it says it is, Trust that apple is running the software it says it is. Security audits help build that trust, but it is not and never will be proof. A three-letter-agency of choice can still walk in and demand they change things without telling anyone. (And while that particular risk is irrelevant to most users, various countries are still opposed to the US having that power over such critical user data.)
  - nl 2y ago
    No, this really isn't right. To quote: verifiable transparency, goes one step further and does away with the hypothetical: security researchers must be able to verify the security and privacy guarantees of Private Cloud Compute, and they must be able to verify that the software that’s running in the PCC production environment is the same as the software they inspected when verifying the guarantees. So how does this work? > The PCC client on the user’s device then encrypts this request directly to the public keys of the PCC nodes that it has first confirmed are valid and cryptographically certified. This provides end-to-end encryption from the user’s device to the validated PCC nodes, ensuring the request cannot be accessed in transit by anything outside those highly protected PCC nodes > Next, we must protect the integrity of the PCC node and prevent any tampering with the keys used by PCC to decrypt user requests. The system uses Secure Boot and Code Signing for an enforceable guarantee that only authorized and cryptographically measured code is executable on the node. All code that can run on the node must be part of a trust cache that has been signed by Apple, approved for that specific PCC node, and loaded by the Secure Enclave such that it cannot be changed or amended at runtime. But why can't a 3-letter agency bypass this? > We designed Private Cloud Compute to ensure that privileged access doesn’t allow anyone to bypass our stateless computation guarantees. > We consider allowing security researchers to verify the end-to-end security and privacy guarantees of Private Cloud Compute to be a critical requirement for ongoing public trust in the system.... When we launch Private Cloud Compute, we’ll take the extraordinary step of making software images of every production build of PCC publicly available for security research. This promise, too, is an enforceable guarantee: user devices will be willing to send data only to PCC nodes that can cryptographically attest to running publicly listed software. So your data will not be sent to node that are not cryptographically attested by third parties. These are pretty strong guarantees, and really make it difficult for Apple to bypass. It's like end-to-end encryption using the Signal protocol: relatively easy to verify it is doing what is claimed, and extraordinarily hard to bypass. Specifically: > The only thing the math tells you is that the server software gave you a correct key. No, this is secure attestation. See for example https://courses.cs.washington.edu/courses/csep590/06wi/finalprojects/bare.pdf https://courses.cs.washington.edu/courses/csep590/06wi/final... which explains it quite well. The weakness of attestation is that you don't know what the root of trust is. But Apple strengthens this by their public inspection and public transparency logs, as well as the target diffusion technique which forces an attack to be very widespread to target a single user. These aren't simple things for a 3LA to work around.
draw_down 2y ago
[dead]
mbesto 2y ago
They already have your private pictures. What difference is it that it's now running AI?
underdeserver 2y ago
Unless you personally validate hardware designs, manufacturing processes, and all software, even when running locally you're trusting many, many people.
blackqueeriroh 2y ago
Ah yes, the unhackable bastion of open-source. …wait, haven’t there been MULTIPLE cases of open-source projects getting hacked in the last couple of years?

seydor 2y ago

> if wanted. Or if someone compels them to

noahtallen 2y ago

I don’t think that’s completely fair. It basically puts Apple in the same bucket as Google or OpenAI. Google obviously tracks everything you do for ads, recommendations, AI, you name it. They don’t even hide it, it’s a core part of their business model. Apple, on the other hand, has made a pretty serious effort to ensure that no employee can access your data on these AI systems. That’s hugely different! They’re going as far as to severely restrict logging and observability and even building and designing their own chips and operating systems. And ensuring that clients will refuse to talk to non-audited systems. Yes, we can’t take Apple’s word for it. But I think the third party audits are a huge part of how we trust, and also verify, that this system will be private. I don’t think it’s far to claim that “Apple knows what you’re doing.” That implies that some one, at some level at Apple can at some point access the data sent from your device to this private cloud. That does not seem to be true. I think another facet of trust here is that a rather big part of Apple’s business model is privacy. They’ve been very successful financially by creating products that generate money in other ways, and it’s very much not necessary or even a sound business idea for them to do something else. While I think it’s fair to be skeptical about the claims without 3rd party verification, I don’t think it’s fair to say that Apple’s approach isn’t better for your data and privacy than openAI or Google. (Which I think is the broad implication — openAI tracks prompts for its own model training, not to resell, so it’s also “only openAI knows what your doing.”)

verisimi 2y ago
It's completely fair, because regardless of third party audits, chips, etc, there are backdoors right along the line, that are going to provide Apple and the government with secret legal access to your data. They can simply go to a secret court, receive a secret judgment, and be authorised to secretly view your data. Does anyone really think this is not already the case? There is no transparency. A licensed third party auditor would not be able to tell you this. We have to operate with the awareness that all data online is already not private - no need to pretend/imagine that Apple's marketing is actually true, and that it is possible to buy online privacy utopia.
- theshrike79 2y ago
  The best protection against "secret orders" is to use mathematics. Build your system so that it can't be decrypted, don't log anything etc. Mullvad has been doing this with VPNs and law enforcement has tested it - there's nothing for them to get. Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone. There's no reason why they wouldn't go to same lengths on their private cloud compute. It's the one thing they can do that Google can't.
  - verisimi 2y ago
    > Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone. Right, but I have no reason to think that this isn't a marketing ploy either, just another story. There is simply no way that Apple is as big as it is, without providing whatever data the government requires. Corporations and governments are not your friend.
    - theshrike79 2y ago
      Apple will obey government orders to give data they have and can access. No government order short of targeting a specific backdoored update to a specific person will allow them to give data they can't access. And if you're doing something that can make a TLA force Apple to create a targeted iOS update just for you, it's not something regular people can or should worry about. Apple keeps normal people safe from mass surveillance, being protected from CIA/NSA required going Full Snowden and it's not a technological problem, you need to change the way you live.

TeMPOraL 2y ago

> unless it's open source and the servers decentralized, you are always trusting SOMEONE Specifically, open-source and self-hostable. Open source doesn't save you if people can't run their own servers, because you never know whether what's in the public repo is the exact same thing that's running on the cloud servers.

nardi 2y ago
This is what the “attestation” bit is supposed to take care of—if it works, which I’m assuming it will, because they’re open sourcing it for security auditing.
jjav 2y ago
> exact same thing that's running on the cloud servers What runs on the servers isn't actually very important. Why? Becuase even if you could somehow know with 100% certainty that what a server runs is the same code you can see, any provider is still subject to all kinds of court orders. What matters is the client code. If you can audit the client code (or better yet, build your own compatible client based on API specs) then you know for sure what the server side sees. If everything is encrypted locally with keys only you control, it doesn't matter what runs on the server.
- flakeoil 2y ago
  But in this use case of AI in the cloud I suppose it's not possible to send encrypted data which only you have the keys to as that makes the data useless and thus no AI processing in the cloud can be made. So the whole point of AI in the cloud vs. AI on device goes away.
dwaite 2y ago
You can by having an attestation of the signed software components up from the secure boot process, and having the client device validate said attestation corresponds to the known public version of each component, and randomize client connections across infrastructure. Other than obvious "open source software isn't perfectly secure" attack scenarios, this would require a non-targeted hardware attack, where the entire infrastructure would need to misinterpret the software or misrepresent the chain of custody. I believe this is one of the protections Apple is attempting to implement here.

detourdog 2y ago

If one has to use tech one has to trust someone. Apple has focused on the individual using computers since inception. They have maintained a consistent message and have a good track record. I will trust them because the alternatives I see are scattered and unfocused.

nl 2y ago

This isn't right. If you trust math you can prove the software is what they say it is. Yes it is work to do this, but this is a big step forward.

ADeerAppeared 2y ago
The only thing the math tells you is that the server software gave you a correct key. It does not tell you how it got that key. A compromised server would send you the key all the same. You still have to trust in the security infrastructure. Trust that Apple is running the hardware it says it is, Trust that apple is running the software it says it is. Security audits help build that trust, but it is not and never will be proof. A three-letter-agency of choice can still walk in and demand they change things without telling anyone. (And while that particular risk is irrelevant to most users, various countries are still opposed to the US having that power over such critical user data.)
- nl 2y ago
  No, this really isn't right. To quote: verifiable transparency, goes one step further and does away with the hypothetical: security researchers must be able to verify the security and privacy guarantees of Private Cloud Compute, and they must be able to verify that the software that’s running in the PCC production environment is the same as the software they inspected when verifying the guarantees. So how does this work? > The PCC client on the user’s device then encrypts this request directly to the public keys of the PCC nodes that it has first confirmed are valid and cryptographically certified. This provides end-to-end encryption from the user’s device to the validated PCC nodes, ensuring the request cannot be accessed in transit by anything outside those highly protected PCC nodes > Next, we must protect the integrity of the PCC node and prevent any tampering with the keys used by PCC to decrypt user requests. The system uses Secure Boot and Code Signing for an enforceable guarantee that only authorized and cryptographically measured code is executable on the node. All code that can run on the node must be part of a trust cache that has been signed by Apple, approved for that specific PCC node, and loaded by the Secure Enclave such that it cannot be changed or amended at runtime. But why can't a 3-letter agency bypass this? > We designed Private Cloud Compute to ensure that privileged access doesn’t allow anyone to bypass our stateless computation guarantees. > We consider allowing security researchers to verify the end-to-end security and privacy guarantees of Private Cloud Compute to be a critical requirement for ongoing public trust in the system.... When we launch Private Cloud Compute, we’ll take the extraordinary step of making software images of every production build of PCC publicly available for security research. This promise, too, is an enforceable guarantee: user devices will be willing to send data only to PCC nodes that can cryptographically attest to running publicly listed software. So your data will not be sent to node that are not cryptographically attested by third parties. These are pretty strong guarantees, and really make it difficult for Apple to bypass. It's like end-to-end encryption using the Signal protocol: relatively easy to verify it is doing what is claimed, and extraordinarily hard to bypass. Specifically: > The only thing the math tells you is that the server software gave you a correct key. No, this is secure attestation. See for example https://courses.cs.washington.edu/courses/csep590/06wi/finalprojects/bare.pdf https://courses.cs.washington.edu/courses/csep590/06wi/final... which explains it quite well. The weakness of attestation is that you don't know what the root of trust is. But Apple strengthens this by their public inspection and public transparency logs, as well as the target diffusion technique which forces an attack to be very widespread to target a single user. These aren't simple things for a 3LA to work around.

draw_down 2y ago

[dead]

mbesto 2y ago

They already have your private pictures. What difference is it that it's now running AI?

underdeserver 2y ago

Unless you personally validate hardware designs, manufacturing processes, and all software, even when running locally you're trusting many, many people.

blackqueeriroh 2y ago

Ah yes, the unhackable bastion of open-source. …wait, haven’t there been MULTIPLE cases of open-source projects getting hacked in the last couple of years?