4 ms·

"We make the hardware and we pinky promise that we will protect your data and will open source part of it" means nothing for privacy. Especially when things lik

"We make the hardware and we pinky promise that we will protect your data and will open source part of it" means nothing for privacy. Especially when things like warrants come into play.

dudus 2y ago
It's a big mental gymnastics to do the same as Google and Microsoft while claiming moral superiority. Apple's thrown stones come back to hunt their glass ceiling.
- cyberpunk 2y ago
  Eh with modern processor features like secure enclaves it's definitely possible to build systems in which the operators CANNOT access the information. (I worked on such a system using SGX for a large car producer, even physical access to the machines/hypervisors/raw memory would not give you access, perhaps the nsa has some keys baked in to extract a session key you may generate inside an enclave, but it would be very surprising if they burned that backdoor on anything as low fruit as this).
  - jeroenhd 2y ago
    SGX has been broken by speculative execution bugs, though. Had something to do with people extracting DRM keys, if I recall correctly, not exactly a nation state attack. Since then, SGX has been removed from modern Intel processors (breaking some Blurays and software products for newer chips in the process). Secure enclave stuff can be used to build a trust relationship if it's designed well, but Apple is the party hosting the service and the one burning the private keys into the chip.
    - cyberpunk 2y ago
      Yep, it was broken a few times but fixed with microcode patches (afaik). It's still a part of the server processors and in wide use already. I'm not saying it's a golden bullet or otherwise infallable, but it sure beats cat /dev/mem by quite some way.
  - bpye 2y ago
    If you produce the hardware you necessarily have access to the signing key to say update the microcode or the firmware. Intel is in the TCB for SGX, but your cloud operator wouldn’t be. In this case Apple is both the hardware manufacturer and the operator.
theshrike79 2y ago
If we can't trust independent audits of code and hardware, what can we trust?
- jeroenhd 2y ago
  I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
  - ReverseCold 2y ago
    > I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
    - jeroenhd 2y ago
      I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
subjectsigma 2y ago
Isn’t this basically what Signal does? Legitimately asking; I thought parts of their server implementation were closed source.
- jeroenhd 2y ago
  I don't think so. Signal regularly stops committing code to the public repos (https://github.com/signalapp/Signal-Server https://github.com/signalapp/Signal-Server) when they're working on some kind of big reveal (cryptocurrency integration and such), but the server code is out there for you to run yourself. Signal has the added benefit that it doesn't need to read what's in the messages you send. It needs some very basic routing information and the rest can be encrypted end to end. With AI stuff, the contents need to be decrypted in the cloud, so the end-to-end protections don't apply.
  - subjectsigma 2y ago
    I meant more regarding their server setup but now that I think about it you are correct, it matters a lot more if the query/message/whatever isn’t encrypted before hitting the cloud.
axoltl 2y ago
How would a warrant work in this case? Using Silicon Data Protection[0] the hash of the currently running firmware (of both the AP _and_ the SEP) is locked into hardware registers in the PKA engine used by the SEP. This hash perturbs the key derivation, and the PKA engine can also attest to the running firmware hash(es) by using an EC key only available to it (they call this BAA, Basic Attestation Authority). iOS won't send any data to a PCC that isn't running a firmware that's been made public in their transparency logs and compute nodes have no way to be debugged in a way that exposes user data[1] And at the end of the day, this is going to give the warrant holder a handful of requests from a specific user? Why wouldn't they use that same warrant to get onto the target's device directly and get that same data plus a ton more? 0: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... 1: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/
- jeroenhd 2y ago
  Apple controls the hardware and the private keys baked into the hardware. If one of their servers can decrypt the payload, they can intercept, duplicate, and decrypt the payload and its response. I'm sure this'll start a long fight between law enforcement and Apple after the first warrant hits and Apple claims it can't comply. Warrants to hack devices are a lot less common and generally harder to obtain. That's why police will send Google warrants for "give us info on every device who has been in a radius of x between y and z time". I'm sure Apple did their very best to protect their users, but I don't think their very best is good enough to warrant this kind of trust. A "secure cloud" solution will also tempt future projects to use the cloud over local processing more, as cloud processing is now readily available. Apple's local processing is a major advantage over the competition but I doubt that'll stay that way if their cloud solution remains this integrated.
  - axoltl 2y ago
    Apple actually does not control the private keys baked into the hardware, see the "Root Cryptographic Keys" section of their security whitepaper: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... Your example indicates a situation where law enforcement does not know which device belongs to their suspect, if they even have one. That's a very different scenario from a targeted "tell us the requests belonging to this individual". Warrants to search a device are extremely common place, otherwise the likes of Grayshift and Cellebrite would not be around. From a threat modeling perspective compromising PCC is high risk (Apple's not just going to comply and the fight will be very public, see the FBI San Bernardino fight) , high effort (Long protracted court case), low reward (I only see requests that are going to get shipped off to the cloud). If I were law enforcement I'd explore every other avenue available to me before I go down that particular rabbit hole which is exactly what this design is intended to achieve.

"We make the hardware and we pinky promise that we will protect your data and will open source part of it" means nothing for privacy. Especially when things like warrants come into play.

dudus 2y ago

It's a big mental gymnastics to do the same as Google and Microsoft while claiming moral superiority. Apple's thrown stones come back to hunt their glass ceiling.

cyberpunk 2y ago
Eh with modern processor features like secure enclaves it's definitely possible to build systems in which the operators CANNOT access the information. (I worked on such a system using SGX for a large car producer, even physical access to the machines/hypervisors/raw memory would not give you access, perhaps the nsa has some keys baked in to extract a session key you may generate inside an enclave, but it would be very surprising if they burned that backdoor on anything as low fruit as this).
- jeroenhd 2y ago
  SGX has been broken by speculative execution bugs, though. Had something to do with people extracting DRM keys, if I recall correctly, not exactly a nation state attack. Since then, SGX has been removed from modern Intel processors (breaking some Blurays and software products for newer chips in the process). Secure enclave stuff can be used to build a trust relationship if it's designed well, but Apple is the party hosting the service and the one burning the private keys into the chip.
  - cyberpunk 2y ago
    Yep, it was broken a few times but fixed with microcode patches (afaik). It's still a part of the server processors and in wide use already. I'm not saying it's a golden bullet or otherwise infallable, but it sure beats cat /dev/mem by quite some way.
- bpye 2y ago
  If you produce the hardware you necessarily have access to the signing key to say update the microcode or the firmware. Intel is in the TCB for SGX, but your cloud operator wouldn’t be. In this case Apple is both the hardware manufacturer and the operator.

theshrike79 2y ago

If we can't trust independent audits of code and hardware, what can we trust?

jeroenhd 2y ago
I will trust independent audits of local code and local hardware. There are still plenty of opportunities for someone to send out malicious patches, but the code running can (and probably will) be analysed by journalists looking for a scoop and security researchers looking for a bug bounty. I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either. Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
- ReverseCold 2y ago
  > I have no idea what code is running on a server I can't access. That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.) The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction. E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740 https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
  - jeroenhd 2y ago
    I do like Apple's attempts to make this stuff better for privacy, but a pinky promise not to leak any information is still just that. Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware. It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.

subjectsigma 2y ago

Isn’t this basically what Signal does? Legitimately asking; I thought parts of their server implementation were closed source.

jeroenhd 2y ago
I don't think so. Signal regularly stops committing code to the public repos (https://github.com/signalapp/Signal-Server https://github.com/signalapp/Signal-Server) when they're working on some kind of big reveal (cryptocurrency integration and such), but the server code is out there for you to run yourself. Signal has the added benefit that it doesn't need to read what's in the messages you send. It needs some very basic routing information and the rest can be encrypted end to end. With AI stuff, the contents need to be decrypted in the cloud, so the end-to-end protections don't apply.
- subjectsigma 2y ago
  I meant more regarding their server setup but now that I think about it you are correct, it matters a lot more if the query/message/whatever isn’t encrypted before hitting the cloud.

axoltl 2y ago

How would a warrant work in this case? Using Silicon Data Protection[0] the hash of the currently running firmware (of both the AP _and_ the SEP) is locked into hardware registers in the PKA engine used by the SEP. This hash perturbs the key derivation, and the PKA engine can also attest to the running firmware hash(es) by using an EC key only available to it (they call this BAA, Basic Attestation Authority). iOS won't send any data to a PCC that isn't running a firmware that's been made public in their transparency logs and compute nodes have no way to be debugged in a way that exposes user data[1] And at the end of the day, this is going to give the warrant holder a handful of requests from a specific user? Why wouldn't they use that same warrant to get onto the target's device directly and get that same data plus a ton more? 0: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... 1: https://security.apple.com/blog/private-cloud-compute/ https://security.apple.com/blog/private-cloud-compute/

jeroenhd 2y ago
Apple controls the hardware and the private keys baked into the hardware. If one of their servers can decrypt the payload, they can intercept, duplicate, and decrypt the payload and its response. I'm sure this'll start a long fight between law enforcement and Apple after the first warrant hits and Apple claims it can't comply. Warrants to hack devices are a lot less common and generally harder to obtain. That's why police will send Google warrants for "give us info on every device who has been in a radius of x between y and z time". I'm sure Apple did their very best to protect their users, but I don't think their very best is good enough to warrant this kind of trust. A "secure cloud" solution will also tempt future projects to use the cloud over local processing more, as cloud processing is now readily available. Apple's local processing is a major advantage over the competition but I doubt that'll stay that way if their cloud solution remains this integrated.
- axoltl 2y ago
  Apple actually does not control the private keys baked into the hardware, see the "Root Cryptographic Keys" section of their security whitepaper: https://help.apple.com/pdf/security/en_US/apple-platform-security-guide.pdf https://help.apple.com/pdf/security/en_US/apple-platform-sec... Your example indicates a situation where law enforcement does not know which device belongs to their suspect, if they even have one. That's a very different scenario from a targeted "tell us the requests belonging to this individual". Warrants to search a device are extremely common place, otherwise the likes of Grayshift and Cellebrite would not be around. From a threat modeling perspective compromising PCC is high risk (Apple's not just going to comply and the fight will be very public, see the FBI San Bernardino fight) , high effort (Long protracted court case), low reward (I only see requests that are going to get shipped off to the cloud). If I were law enforcement I'd explore every other avenue available to me before I go down that particular rabbit hole which is exactly what this design is intended to achieve.