5 ms·

> I see no real difference between 2 and 3. This #2, so-called "Private Cloud Compute", is not the same as iCloud. And certainly not the same as sending queri

> I see no real difference between 2 and 3. This #2, so-called "Private Cloud Compute", is not the same as iCloud. And certainly not the same as sending queries to OpenAI. Quoting: “With Private Cloud Compute, Apple Intelligence can flex and scale its computational capacity and draw on larger, server-based models for more complex requests. These models run on servers powered by Apple silicon, providing a foundation that allows Apple to ensure that data is never retained or exposed.“ “Independent experts can inspect the code that runs on Apple silicon servers to verify privacy, and Private Cloud Compute cryptographically ensures that iPhone, iPad, and Mac do not talk to a server unless its software has been publicly logged for inspection.” “Apple Intelligence with Private Cloud Compute sets a new standard for privacy in AI, unlocking intelligence users can trust.”

jeroenhd 2y ago
"We make the hardware and we pinky promise that we will protect your data and will open source part of it" means nothing for privacy. Especially when things like warrants come into play.
- dudus 2y ago
  It's a big mental gymnastics to do the same as Google and Microsoft while claiming moral superiority. Apple's thrown stones come back to hunt their glass ceiling.
  - cyberpunk 2y ago
    Eh with modern processor features like secure enclaves it's definitely possible to build systems in which the operators CANNOT access the information. (I worked on such a system using SGX for a large car producer, even physical access to the machines/hypervisors/raw memory would not give you access, perhaps the nsa has some keys baked in to extract a session key you may generate inside an enclave, but it would be very surprising if they burned that backdoor on anything as low fruit as this).
    - jeroenhd 2y ago
      SGX has been broken by speculative execution bugs, though. Had something to do with people extracting DRM keys, if I recall correctly, not exactly a nation state attack. Since then, SGX has been removed from modern Intel processors (breaking some Blurays and software products for newer chips in the process). Secure enclave stuff can be used to build a trust relationship if it's designed well, but Apple is the party hosting the service and the one burning the private keys into the chip.
      - cyberpunk 2y ago
        Yep, it was broken a few times but fixed with microcode patches (afaik). It's still a part of the server processors and in wide use already. I'm not saying it's a golden bullet or otherwise infallable, but it sure beats cat /dev/mem by quite some way.
    - bpye 2y ago
      If you produce the hardware you necessarily have access to the signing key to say update the microcode or the firmware. Intel is in the TCB for SGX, but your cloud operator wouldn’t be. In this case Apple is both the hardware manufacturer and the operator.
- theshrike79 2y ago
  If we can't trust independent audits of code and hardware, what can we trust?
  - jeroenhd 2y ago
    I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
    - ReverseCold 2y ago
      > I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
      - jeroenhd 2y ago
        I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
- subjectsigma 2y ago
  Isn’t this basically what Signal does? Legitimately asking; I thought parts of their server implementation were closed source.
  - jeroenhd 2y ago
    I don't think so. Signal regularly stops committing code to the public repos (https://github.com/signalapp/Signal-Server https://github.com/signalapp/Signal-Server) when they're working on some kind of big reveal (cryptocurrency integration and such), but the server code is out there for you to run yourself. Signal has the added benefit that it doesn't need to read what's in the messages you send. It needs some very basic routing information and the rest can be encrypted end to end. With AI stuff, the contents need to be decrypted in the cloud, so the end-to-end protections don't apply.
    - subjectsigma 2y ago
      I meant more regarding their server setup but now that I think about it you are correct, it matters a lot more if the query/message/whatever isn’t encrypted before hitting the cloud.
- axoltl 2y ago
  How would a warrant work in this case? Using Silicon Data Protection[0] the hash of the currently running firmware (of both the AP _and_ the SEP) is locked into hardware registers in the PKA engine used by the SEP. This hash perturbs the key derivation, and the PKA engine can also attest to the running firmware hash(es) by using an EC key only available to it (they call this BAA, Basic Attestation Authority). iOS won't send any data to a PCC that isn't running a firmware that's been made public in their transparency logs and compute nodes have no way to be debugged in a way that exposes user data[1] And at the end of the day, this is going to give the warrant holder a handful of requests from a specific user? Why wouldn't they use that same warrant to get onto the target's device directly and get that same data plus a ton more? 0: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... 1: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/
  - jeroenhd 2y ago
    Apple controls the hardware and the private keys baked into the hardware. If one of their servers can decrypt the payload, they can intercept, duplicate, and decrypt the payload and its response. I'm sure this'll start a long fight between law enforcement and Apple after the first warrant hits and Apple claims it can't comply. Warrants to hack devices are a lot less common and generally harder to obtain. That's why police will send Google warrants for "give us info on every device who has been in a radius of x between y and z time". I'm sure Apple did their very best to protect their users, but I don't think their very best is good enough to warrant this kind of trust. A "secure cloud" solution will also tempt future projects to use the cloud over local processing more, as cloud processing is now readily available. Apple's local processing is a major advantage over the competition but I doubt that'll stay that way if their cloud solution remains this integrated.
    - axoltl 2y ago
      Apple actually does not control the private keys baked into the hardware, see the "Root Cryptographic Keys" section of their security whitepaper: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... Your example indicates a situation where law enforcement does not know which device belongs to their suspect, if they even have one. That's a very different scenario from a targeted "tell us the requests belonging to this individual". Warrants to search a device are extremely common place, otherwise the likes of Grayshift and Cellebrite would not be around. From a threat modeling perspective compromising PCC is high risk (Apple's not just going to comply and the fight will be very public, see the FBI San Bernardino fight) , high effort (Long protracted court case), low reward (I only see requests that are going to get shipped off to the cloud). If I were law enforcement I'd explore every other avenue available to me before I go down that particular rabbit hole which is exactly what this design is intended to achieve.
StrLght 2y ago
That's too many words with surprisingly little meaning. I'd suggest to wait for more technical details and to treat this as marketing until then.
- _ye0b 2y ago
  You can read more here: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/ But in summary 1. The servers run on Apple Silicon hardware which have fancier security features 2. Software is open source 3. iOS verifies that the server is actually running that open source software before talking to it 4. This is insane privacy for AI The security features are meant to prevent the server operator (Apple) from being able to access data that's being processed in their farm. The idea is that with that + E2E encryption, it should be way closer to on-device processing in terms of privacy and security
  - StrLght 2y ago
    Thanks! That's great and sounds like they're really trying to go as far as possible with it. Here's also a great summary from Matthew Green: https://x.com/matthew_d_green/status/1800291897245835616 https://x.com/matthew_d_green/status/1800291897245835616
SoftTalker 2y ago
Yes, that's all well and good but assumes no mistakes and no National Security letters ordering them to describe it that way and no changes of control or business strategy at some point in the future. Once the data is out of your possession it's out of your control.
- mostlysimilar 2y ago
  Not everyone has nation states in their threat models. I want privacy from corporations / surveillance capitalism, not the US government. Apple's privacy promises are focused on keeping my data out of the hands of bad actors like Google etc. and that's more than enough for me.
  - magicalist 2y ago
    A threat model of Google getting your email revision makes these statements sillier. TLS and existing privacy policies are sufficient.
- theshrike79 2y ago
  There are VERY few things that can keep your information safe if a TLA wants it. You need to go full Edward Snowden with phones in faraday cages and typing passwords under a sheet -levels of paranoia to be fully safe. Drow "nation state is after me" from the threat model and you'll be a lot happier.
  - transpute 2y ago
    > nation state ... threat model The history of tech is the history of falling costs with mass production. Expensive TLA surveillance tech for nation states can become broadly accessible, e.g. through-wall WiFi radar sensing sold to millions via IEEE 802.11bf WiFi 7 Sensing in NPU/AI PCs [1], or USB implant cables [2] with a few zeros lopped off the TLA price. Instead of adversary motives, threat models can be based on adversary costs. As adversary costs fall, threat models need to evolve. [1] https://www.technologyreview.com/2024/02/27/1088154/wifi-sensing-tracking-movements/ https://www.technologyreview.com/2024/02/27/1088154/wifi-sen... [2] https://shop.hak5.org/products/omg-cable https://shop.hak5.org/products/omg-cable

jeroenhd 2y ago

"We make the hardware and we pinky promise that we will protect your data and will open source part of it" means nothing for privacy. Especially when things like warrants come into play.

dudus 2y ago
It's a big mental gymnastics to do the same as Google and Microsoft while claiming moral superiority. Apple's thrown stones come back to hunt their glass ceiling.
- cyberpunk 2y ago
  Eh with modern processor features like secure enclaves it's definitely possible to build systems in which the operators CANNOT access the information. (I worked on such a system using SGX for a large car producer, even physical access to the machines/hypervisors/raw memory would not give you access, perhaps the nsa has some keys baked in to extract a session key you may generate inside an enclave, but it would be very surprising if they burned that backdoor on anything as low fruit as this).
  - jeroenhd 2y ago
    SGX has been broken by speculative execution bugs, though. Had something to do with people extracting DRM keys, if I recall correctly, not exactly a nation state attack. Since then, SGX has been removed from modern Intel processors (breaking some Blurays and software products for newer chips in the process). Secure enclave stuff can be used to build a trust relationship if it's designed well, but Apple is the party hosting the service and the one burning the private keys into the chip.
    - cyberpunk 2y ago
      Yep, it was broken a few times but fixed with microcode patches (afaik). It's still a part of the server processors and in wide use already. I'm not saying it's a golden bullet or otherwise infallable, but it sure beats cat /dev/mem by quite some way.
  - bpye 2y ago
    If you produce the hardware you necessarily have access to the signing key to say update the microcode or the firmware. Intel is in the TCB for SGX, but your cloud operator wouldn’t be. In this case Apple is both the hardware manufacturer and the operator.
theshrike79 2y ago
If we can't trust independent audits of code and hardware, what can we trust?
- jeroenhd 2y ago
  I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
  - ReverseCold 2y ago
    > I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
    - jeroenhd 2y ago
      I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
subjectsigma 2y ago
Isn’t this basically what Signal does? Legitimately asking; I thought parts of their server implementation were closed source.
- jeroenhd 2y ago
  I don't think so. Signal regularly stops committing code to the public repos (https://github.com/signalapp/Signal-Server https://github.com/signalapp/Signal-Server) when they're working on some kind of big reveal (cryptocurrency integration and such), but the server code is out there for you to run yourself. Signal has the added benefit that it doesn't need to read what's in the messages you send. It needs some very basic routing information and the rest can be encrypted end to end. With AI stuff, the contents need to be decrypted in the cloud, so the end-to-end protections don't apply.
  - subjectsigma 2y ago
    I meant more regarding their server setup but now that I think about it you are correct, it matters a lot more if the query/message/whatever isn’t encrypted before hitting the cloud.
axoltl 2y ago
How would a warrant work in this case? Using Silicon Data Protection[0] the hash of the currently running firmware (of both the AP _and_ the SEP) is locked into hardware registers in the PKA engine used by the SEP. This hash perturbs the key derivation, and the PKA engine can also attest to the running firmware hash(es) by using an EC key only available to it (they call this BAA, Basic Attestation Authority). iOS won't send any data to a PCC that isn't running a firmware that's been made public in their transparency logs and compute nodes have no way to be debugged in a way that exposes user data[1] And at the end of the day, this is going to give the warrant holder a handful of requests from a specific user? Why wouldn't they use that same warrant to get onto the target's device directly and get that same data plus a ton more? 0: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... 1: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/
- jeroenhd 2y ago
  Apple controls the hardware and the private keys baked into the hardware. If one of their servers can decrypt the payload, they can intercept, duplicate, and decrypt the payload and its response. I'm sure this'll start a long fight between law enforcement and Apple after the first warrant hits and Apple claims it can't comply. Warrants to hack devices are a lot less common and generally harder to obtain. That's why police will send Google warrants for "give us info on every device who has been in a radius of x between y and z time". I'm sure Apple did their very best to protect their users, but I don't think their very best is good enough to warrant this kind of trust. A "secure cloud" solution will also tempt future projects to use the cloud over local processing more, as cloud processing is now readily available. Apple's local processing is a major advantage over the competition but I doubt that'll stay that way if their cloud solution remains this integrated.
  - axoltl 2y ago
    Apple actually does not control the private keys baked into the hardware, see the "Root Cryptographic Keys" section of their security whitepaper: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... Your example indicates a situation where law enforcement does not know which device belongs to their suspect, if they even have one. That's a very different scenario from a targeted "tell us the requests belonging to this individual". Warrants to search a device are extremely common place, otherwise the likes of Grayshift and Cellebrite would not be around. From a threat modeling perspective compromising PCC is high risk (Apple's not just going to comply and the fight will be very public, see the FBI San Bernardino fight) , high effort (Long protracted court case), low reward (I only see requests that are going to get shipped off to the cloud). If I were law enforcement I'd explore every other avenue available to me before I go down that particular rabbit hole which is exactly what this design is intended to achieve.

StrLght 2y ago

That's too many words with surprisingly little meaning. I'd suggest to wait for more technical details and to treat this as marketing until then.

_ye0b 2y ago
You can read more here: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/ But in summary 1. The servers run on Apple Silicon hardware which have fancier security features 2. Software is open source 3. iOS verifies that the server is actually running that open source software before talking to it 4. This is insane privacy for AI The security features are meant to prevent the server operator (Apple) from being able to access data that's being processed in their farm. The idea is that with that + E2E encryption, it should be way closer to on-device processing in terms of privacy and security
- StrLght 2y ago
  Thanks! That's great and sounds like they're really trying to go as far as possible with it. Here's also a great summary from Matthew Green: https://x.com/matthew_d_green/status/1800291897245835616 https://x.com/matthew_d_green/status/1800291897245835616

SoftTalker 2y ago

Yes, that's all well and good but assumes no mistakes and no National Security letters ordering them to describe it that way and no changes of control or business strategy at some point in the future. Once the data is out of your possession it's out of your control.

mostlysimilar 2y ago
Not everyone has nation states in their threat models. I want privacy from corporations / surveillance capitalism, not the US government. Apple's privacy promises are focused on keeping my data out of the hands of bad actors like Google etc. and that's more than enough for me.
- magicalist 2y ago
  A threat model of Google getting your email revision makes these statements sillier. TLS and existing privacy policies are sufficient.
theshrike79 2y ago
There are VERY few things that can keep your information safe if a TLA wants it. You need to go full Edward Snowden with phones in faraday cages and typing passwords under a sheet -levels of paranoia to be fully safe. Drow "nation state is after me" from the threat model and you'll be a lot happier.
- transpute 2y ago
  > nation state ... threat model The history of tech is the history of falling costs with mass production. Expensive TLA surveillance tech for nation states can become broadly accessible, e.g. through-wall WiFi radar sensing sold to millions via IEEE 802.11bf WiFi 7 Sensing in NPU/AI PCs [1], or USB implant cables [2] with a few zeros lopped off the TLA price. Instead of adversary motives, threat models can be based on adversary costs. As adversary costs fall, threat models need to evolve. [1] https://www.technologyreview.com/2024/02/27/1088154/wifi-sensing-tracking-movements/ https://www.technologyreview.com/2024/02/27/1088154/wifi-sen... [2] https://shop.hak5.org/products/omg-cable https://shop.hak5.org/products/omg-cable