25 ms·
Private Cloud Compute: A new frontier for AI privacy in the cloud
- ethbr1 2y agoThis entire platform is the first time I've strategically considered realigning the majority of my use to Apple. Airtag anonymity was pretty cool, technically speaking, but a peripheral use case for me. To me, PCC is a well-reasoned, surprisingly customer-centric response to the fact that due to (processing, storage, battery) limitations not all useful models can be run on-device. And they tried to build a privacy architecture before widely deploying it, instead of post-hoc bolting it on. >> 4. Non-targetability. An attacker should not be able to attempt to compromise personal data that belongs to specific, targeted Private Cloud Compute users without attempting a broad compromise of the entire PCC system. This must hold true even for exceptionally sophisticated attackers who can attempt physical attacks on PCC nodes in the supply chain or attempt to obtain malicious access to PCC data centers. Oof. That's a pretty damn specific (literally) attacker, and it's impressive that made it into their threat model. And neat use of onion-style encryption to expose the bare minimum necessary for routing, before the request reaches its target node. Also [0] >> For example, the [PCC node OS] doesn’t even include a general-purpose logging mechanism. Instead, only pre-specified, structured, and audited logs and metrics can leave the node, and multiple independent layers of review help prevent user data from accidentally being exposed through these mechanisms. My condolences to Apple SREs, between this and the other privacy guarantees. >> Our commitment to verifiable transparency includes: (1) Publishing the measurements of all code running on PCC in an append-only and cryptographically tamper-proof transparency log. (2) Making the log and associated binary software images publicly available for inspection and validation by privacy and security experts. (3) Publishing and maintaining an official set of tools for researchers analyzing PCC node software. (4) Rewarding important research findings through the Apple Security Bounty program. So binary-only for majority, except the following: >> While we’re publishing the binary images of every production PCC build, to further aid research we will periodically also publish a subset of the security-critical PCC source code. >> In a first for any Apple platform, PCC images will include the sepOS firmware and the iBoot bootloader in plaintext, making it easier than ever for researchers to study these critical components. [0] Oblivious HTTP, https://www.rfc-editor.org/rfc/rfc9458 https://www.rfc-editor.org/rfc/rfc9458
- transpute 2y agoIt's very encouraging. Another good step in this direction would be publishing a list of all on-device Apple software (including Spotlight models for image analysis) and details of any information that is sent to Apple, along with opt-out instructions via device Settings or Apple Configurator MDM profiles. Apple does publish a list of network ports and servers, so that network traffic can be permitted for specific services. The list is complicated by 3rd-party CDNs, but can be made to work with dnsmasq and ipset, "Use Apple products on enterprise networks", https://support.apple.com/en-us/101555 https://support.apple.com/en-us/101555
- manquer 2y ago> Oof. That's a pretty damn specific (literally) attacker, and it's impressive that made it into their threat model. How so ? There are any number of state and state sponsored attackers who it should apply it including china, North Korea , Russia , Israel as nation states and their various affiliates like NSO group . Even if NSA its related entities are going to be notably absent. If your threat model includes unfriendly nation state actors then the security depends on security at NSA and less on Apple, they have all your data anyway. If nation state actors are interested in you, no smartphone that is not fully open source on both hardware and OS side that has been independently verified by multiple reviewers is worth it, i.e. no phone in the market today, everything else is tradeoff for convenience for risk, the degree of each is quite subjective to each individual. For the rest of us, the threat model is advertisers, identity thieves, scammers and spammers and now AI companies using it for training. Apple will protect against other advertisers insofar to grow their own ad platform , they already sell searches to Google for $20B/year and there is no knowing the details of the OpenAI deal on what kind of data will be shared.
- thomasahle 2y agoDid Apple say anything about what training data they used for their generative image models?
- jeffbee 2y agoA lot of this sounds like Apple has been 10-20 years behind the state of the art and now wants to tell you that they partially caught up. Verifiable hardware roots of trust and end-to-end software supply chain integrity are things that have existed for a while. The interesting part doesn't come until the end where they promise to publish system images for inspection.
- ethbr1 2y agoDo you have analogous search terms for Microsoft, Alphabet, Google, and Amazon's approaches? Your comment makes me curious on how guarantee-to-guarantee looks (and associated architectures).
- candiddevmike 2y agoMost of the stuff in the blog post reads like common security precautions: don't run as root, stateless immutable nodes, use secure boot, etc. All wrapped up in some Apple marketing pizzazz.
- ignoramous 2y ago> common security precautions ... marketing pizzazz. If it were this common, Meta, Google, and others would have announced or launched something similar for its consumer apps/services; I can't seem to recall anything of note.
- bowmessage 2y agohttps://cloud.google.com/docs/security/binary-authorization-for-borg https://cloud.google.com/docs/security/binary-authorization-... is one example
- threeseed 2y agoApple's system goes further by having incoming requests choose and verify a server and then encrypt itself using the public key of the node to prevent MITM attacks. And a one-time credential to prevent replay attacks. As well as minor things like obfuscating IP addresses, metadata etc.
- candiddevmike 2y agoDid Apple need to license the phrase Core OS like iOS?
- thirdhaf 2y agoThat phrase distinguishes the internal group responsible for that part of the architecture, don’t think it’s a marketing term.
- Shank 2y ago> In a first for any Apple platform, PCC images will include the sepOS firmware and the iBoot bootloader in plaintext, making it easier than ever for researchers to study these critical components. Yes! > Software will be published within 90 days of inclusion in the log, or after relevant software updates are available, whichever is sooner. I think this theoretically leaves a 90-day maximum gap between publishing vulnerable software and potential-for-discovery. I sincerely hope that the actual availability of images is closer to instant than the maximum, though.
- gigel82 2y agoWell, a 89-day "update-and-revert" schedule will take care of those pesky auditors asking too many questions about NSA's backdoor or CCP's backdoor and all that.
- gpm 2y agoNo, because the log of what source was used will still show the backdoored version, and you can't unpublish the information that it was used. Reverting doesn't solve the problem that people will be able to say "this software was attested 90 days ago and it hasn't been released". If you're trying to do a quiet backdoor and you have the power to compel Apple to assist, the route to take is to simply misuse the keys that are supposed to only go into hardware for attestation, and instead simply use them to forge messages attesting to be running software on hardware that you aren't. Or just find a bug in the software stack that gives you RCE and use it
- brookst 2y ago> simply use them to forge messages attesting to be running software on hardware that you aren't Well, your messages have to be congruent with the expected messages from the real hardware, and your fake hardware has to register with the real load balancers to receive user requests. > RCE That’s probably the best attack vector, and presumably why Apple is only making binary executables available. Not that that stops RCE. But even then you can’t pick and choose the users whose data you compromise. It’s still a sev0 problem, but less exploitable for the goals of nation states so less likely to be heavily invested in for exploiting.
- Havoc 2y agoSounds good. Still won’t send anything sensitive there but I appreciate the effort and direction, especially when current industry trend seems to be fuck you were rewriting our TOS to take your data.
- m-s-y 2y agoApple’s doing this specifically to avoid the possibility of what you’re describing. The transparency & architecture together are intended to be more than enough to publicly detect any major retooling of the system.
- advael 2y agoI really want to see this OS, and have cautious optimism that this could be the first time we'll see a big tech company actually provide an auditable security guarantee! I think depending on how this plays out, Apple might manage to earn some of the trust its users have in it, which would be pretty cool! But even cooler will be if we get full chain-of-custody audits, which I think will have to entail opening up some other bits of their stack In particular, the cloud OS being open-source, if they make good on that commitment, will be incredibly valuable. My main concern right now is that if virtualization is employed in their actual deployment, there could be a backdoor that passes keys from secure enclaves in still-proprietary parts of the OSes running on user devices to a hypervisor we didn't audit that can access the containers. Surely people with more security expertise than me will have even better questions. Maybe Apple will be responsive to feedback from researchers and this could lead to more of this toolchain being auditable. But even if we can't verify that their sanctioned use case is secure, the cloud OS could be a great step forward in secure inference and secure clouds, which people could independently host or build an independent derivative of The worst case is still that they just don't actually do it, but it seems reasonably likely they'll follow through on at least that, and then the worst case becomes "Super informative open-source codebase for secure computing at scale just dropped" which is a great thing no matter how the other stuff goes
- transpute 2y ago> even if we can't verify that their sanctioned use case is secure, the cloud OS could be a great step forward in secure inference and secure clouds, which people could independently host or build an independent derivative of Yes, the tech industry loves to copy Apple :) Asahi Linux has a good overview of on-device boot chain security, https://github.com/AsahiLinux/docs/wiki/Apple-Platform-Security-Crash-Course https://github.com/AsahiLinux/docs/wiki/Apple-Platform-Secur... > My main concern right now is that if virtualization is employed in their actual deployment, there could be a backdoor that passes keys from secure enclaves in still-proprietary parts of the OSes running on user devices to a hypervisor we didn't audit that can access the containers. We’ll release a PCC Virtual Research Environment: a set of tools and images that simulate a PCC node on a Mac with Apple silicon, and that can boot a version of PCC software minimally modified for successful virtualization. This seems to imply that PCC nodes are bare-metal. Could a PCC node be simulated on iPad Pro with M4 Apple Silicon?
- telepathy 2y ago[flagged]
- loteck 2y agoSome good comments on this from cryptographer Matt Green here: https://x.com/matthew_d_green/status/1800291897245835616?t=CcBMWojQZYI0RnnS_gVWzA&s=19 https://x.com/matthew_d_green/status/1800291897245835616?t=C... (I wonder if Matt realizes nobody can read his tweets without a X account? Use BlueSky or Masto man) Edit: here's his thread combined https://threadreaderapp.com/thread/1800291897245835616.html?utm_campaign=topunroll https://threadreaderapp.com/thread/1800291897245835616.html?...
- transpute 2y agoThanks for the link. > As best I can tell, Apple does not have explicit plans to announce when your data is going off-device for to Private Compute. You won't opt into this, you won't necessarily even be told it's happening. It will just happen. Magically. Presumably it will be possible to opt out of AI features entirely, i.e. both on-device and off-device? Why would a device vendor not have an option for on-device AI only? iOS 17 AI features can be used today without iCloud. Hopefully Apple uses a unique domain (e.g. *.pcc.apple.com) that can be filtered at the network level.
- azinman2 2y agoYou would have to activate a clearly LLM-powered software feature and have internet access. I don't know if settings will appear to disable this, but you could imagine it would be the case. This isn't just siphoning off all your data at random.
- transpute 2y agoWould Spotlight be considered a "clearly LLM-powered software feature"? Will there be an option for "non-AI Spotlight"? Disabling dozens of software features, or identifying all apps which might use LLM services, is a daunting proposition. It would be good to have a PCC kill switch, which makes opt-in usage meaningful, rather than forced.
- chefandy 2y ago
- cherioo 2y agoCan some ELI5 how remote attestation is supposed to work? It feels like asking a remote endpoint “are you who you say you are”. What’s stopping remote endpoint always responding “yes”
- davidczech 2y agoServers send signed statements describing its state, and clients make the yes/no determination.
- transpute 2y ago> What’s stopping remote endpoint always responding “yes” It requires a small, trusted remote observer hardware component, e.g. TCG TPM/DICE, Apple Secure Enclave, Google OpenTitan, Microsoft Pluton. 2021 literature review, https://arxiv.org/abs/2105.02466 https://arxiv.org/abs/2105.02466 2022 HN thread on remote attestation, https://news.ycombinator.com/item?id=32282305 https://news.ycombinator.com/item?id=32282305
- wyes 2y agoThey rely on Trusted Execution Environments and the fact that hash functions are one-way functions. Verifier -> requests a Prover to attest its software state Prover -> goes into RoT, verifies authenticity of Verifier (and request), computes hash of attested memory region, sends hash digest Verifier -> receives digest and compares to known hash > What’s stopping remote endpoint always responding “yes” The attestation code is inside of a RoT, so a bad actor shouldn't be able to call this code, only callable by receiving a request from a Verifier
- GrantMoyer 2y agoMy understanding is that it's similar to TLS authentication. The remote endpoint has special hardware which keeps secret signing keys (similar to a TLS server's signing keys). The hardware refuses to reveal the private keys, but will sign certain payloads under certain conditions. In addition, Intel or AMD or whoever also has super duper mega secret master keys (similar to a CA's signing keys), which they use to sign the device's signing keys. The certificate signing the device keys is also stored on the device. So, each time the endpoint is asked to attest its software, it says yes and signs its response with its keys, and it also sends a certificate showing its keys are signed by the master key. That way, the client knows the special hardware really said yes and that Intel or AMD or whoever said that particular special hardware is legit.
- rmbyrro 2y agoMost ironic thing is they abbreviate this as "PCC". (reads chinese communist party in many languages) The absolute worst acronym for anything even remotely related to personal privacy.
- transpute 2y agoInverted acronym?
- deleted 2y ago[deleted]
- zer00eyz 2y agoI have a big question here. Who is this for? Dont get me wrong I think it's a great effort. This is some A+ nerd stuff right here. It's speaking my languge. But Im just going to figure out how to turn off "calls home". Cause I dont want it doing this at all. Is this speaking to me so I tell others "apple is the most secure option"? I don't want to tell others "linux" because I don't want to do tech support for that. At this point I feel like an old man shouting "Dam you keep your hands off my data".
- wmf 2y agoWhat if you can't turn it off and this extreme security is the justification for why?
- deleted 2y ago[deleted]
- al_borland 2y ago
- SirensOfTitan 2y agoWhat I'm most curious about here is if a state actor comes to Apple with a subpoena and compels them to release information on an individual, what would Apple be able to release? ... I suppose this is ultimately a question that will be tested sooner or later in the US.
- ricardobeat 2y agoUnless their statements regarding the design of these systems are blatantly false, or they are forced to add data collectors on purpose to target individuals, the answer is close to nothing. You can opt into full E2E encryption [1] which makes it nothing, presumably at the cost of some convenience features. [1] https://support.apple.com/en-us/108756 https://support.apple.com/en-us/108756
- onesociety2022 2y agoPCC exists because full E2E is not feasible for these use cases. The LLM has to take your personal data (context window and prompt) to process it.
- deleted 2y ago[deleted]
- riscy 2y agoI mean it was famously tested in 2015 after the San Bernardino attack. Apple didn’t back down [1] and later sued the company who sold the zero-day to the govt to unlock the phone [2]. [1] https://en.m.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_dispute https://en.m.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption... [2] https://www.washingtonpost.com/technology/2021/04/14/azimuth-san-bernardino-apple-iphone-fbi/ https://www.washingtonpost.com/technology/2021/04/14/azimuth...
- asadotzler 2y agoAlso famously tested (and failed) much more recently. https://arstechnica.com/tech-policy/2023/12/apple-admits-to-secretly-giving-governments-push-notification-data/ https://arstechnica.com/tech-policy/2023/12/apple-admits-to-... Apple shills are the worst.
- deleted 2y ago[deleted]
- yla92 2y ago> And finally, we used Swift on Server to build a new Machine Learning stack specifically for hosting our cloud-based foundation model. Interesting to see Swift on Server here! https://www.swift.org/documentation/server/ https://www.swift.org/documentation/server/
- jaydeegee 2y agoOutside of all the security aspects which look to be handled quite well on the surface I do enjoy that the client mainframe architecture is still a staple of computing.
- bayareabadboy 2y agoWhat are the longer term implications that Apple is doing this on their own hardware and not Nvidia? This seems like a big thing to me, an idiot.
- wmf 2y agoIf you're one of the richest companies in history you can "simply" invest 15 years into developing your own chips instead of buying Nvidia GPUs.
- transpute 2y ago> simply invest 15 years into developing your own chips instead of buying Nvidia GPUs https://www.notebookcheck.net/Apple-and-Imagination-strike-GPU-deal-after-briefly-parting-ways.448976.0.html https://www.notebookcheck.net/Apple-and-Imagination-strike-G... Following the loss of Apple, easily its biggest client, Imagination was bought out by a Chinese-based investment group. Apple subsequently released its first in-house designed mobile GPU as part of the A11 Bionic SoC that powered the iPhone X.. The new “multi-year license agreement” gives Apple official access to much wider range of Imagination’s mobile GPU IP as well as its AI technologies. The A11 Bionic also included the first neural processing engine in an iPhone https://9to5mac.com/2020/01/01/apple-imagination-agreement/ https://9to5mac.com/2020/01/01/apple-imagination-agreement/ Apple described Imagination’s characterizations as misleading while hiring Imagination employees to work for Apple’s GPU team in the same community.
- j0e1 2y ago> The Apple Security Bounty will reward research findings in the entire Private Cloud Compute software stack — with especially significant payouts for any issues that undermine our privacy claims. Let the games begin!
- piccirello 2y ago> The Secure Enclave randomizes the data volume’s encryption keys on every reboot and does not persist these random keys, ensuring that data written to the data volume cannot be retained across reboot. In other words, there is an enforceable guarantee that the data volume is cryptographically erased every time the PCC node’s Secure Enclave Processor reboots.
- Timber-6539 2y agoFeels like an uptime screenshot would be appropriate here
- transpute 2y agoPCC node execution should be per-transaction, i.e. relatively short lived.
- wmf 2y agoThe server can't afford to do one transaction then reboot.
- transpute 2y agoIntel and AMD server processors can use DRTM late launch for fast attested restart, https://www.semanticscholar.org/paper/An-Execution-Infrastructure-for-TCB-Minimization-McCune-Parno/89d0ae6690a601ca54fec6c339b561f00a7fbfb1 https://www.semanticscholar.org/paper/An-Execution-Infrastru.... If future Apple Silicon processors can support late launch, then PCC nodes can reduce intermingling of data from multiple customer transactions. > The server can't afford What reboot frequency is affordable for PCC nodes?
- throwaway369 2y ago[Deleted]
- zie 2y agoWell the options from China's perspective is: Come to the table and meet some/all of our demands or stop doing business here. Since Apple devices are now on the Chinese Governments poopy list, I assume Apple is only meeting some, not all of China's demands. I assume if Apple did everything the Chinese govt wanted, they wouldn't be on the poopy list. Personally I see being on the Chinese govt poopy list as an endorsement that it's probably a net positive for privacy and security compared to those not on the list. :) Around WhatsApp, it's probably part of the whole compromise mess above. WhatsApp now does E2E and that's something China is not a fan of, so it's probably China's doing that it's not in the app store in China any more. Apple is just following the laws China forces them to follow. It should be noted I've never been to China(yet) and have zero 1st hand knowledge.
- astrange 2y agoIt's a silly oversimplification that nothing in China is ever allowed to have privacy ever. China has privacy/data protection laws just like other countries do. Even an authoritarian government doesn't want other random private actors getting to see everything.
- zie 2y agoI agree, but I was talking specifically about the govt. The govt basically requires total access doesn't it? I mean every govt basically wants it, and the US has tried many times, but so far hasn't quite gotten complete access everywhere.
- m3kw9 2y agoI wonder how they will do this in china?
- mlindner 2y agoApple already runs China-only software on their devices, I suppose it just won't run there.
- solarkraft 2y agoI was sceptical of the announcement, but this actually sounds really well thought out. One key part though will be the remote attestation that the servers are actually running what they say they're running. Without any access to the servers, how do we do that? Am I correctly expecting that that part remains a "trust me bro" situation?
- wyes 2y agoAttestation will run on the RoT. >While we’re publishing the binary images of every production PCC build, to further aid research we will periodically also publish a subset of the security-critical PCC source code. I expect that they'll publish the attestation source code. But, basically what will happen is the Verifier will request a certain memory region to be attested, then that region will be hashed and the digest will be sent back to the Verifier. If the memory is different from what is expected, the hash digest will NOT match.
- deleted 2y ago[deleted]
- ein0p 2y agoIt is not possible for this to be fully private in the United States because the government not only can force Apple to open up the kimono, it can also forbid it to talk about it. There’s not really anything Apple can do to work around this “limitation”. Thank your “representative” for extending the PATRIOT Act when you get a chance.
- paradite 2y agoSlightly off-topic, "open up the kimono" sounds disturbing and creepy to me as an Asian. I suspect I'm not alone in this.
- ein0p 2y agoThat’s even better. I do think it’s disturbing and creepy when someone goes through my private data without my knowledge.
- digging 2y agoThat's not what they meant or what the phrase means. It's mildly racist and misogynist, and the kinds of discomfort it elicits are not the kind that will make people trust you, the user of the phrase.
- dxbednarczyk 2y agoSome share your sentiment. https://www.npr.org/sections/codeswitch/2014/11/02/360479744/why-corporate-executives-talk-about-opening-their-kimonos https://www.npr.org/sections/codeswitch/2014/11/02/360479744...
- clipjokingly 2y agoIs it possible to have zero knowledge AI?
- wslh 2y agoYes, the issue is that they are really slow.
- rjeli 2y agoZKML is actually not horrible, probably only 100-1000x overhead atm. Unfortunately it doesn’t solve the problem, you would need FHE which has much higher overhead
- tharant 2y agoFHE? I, a noob, assume that acronym maybe has something to do with homomorphic encryption? Also, got any links for interesting ZKML papers/projects?
- ramesh31 2y agoHere's the answer to the "what's taking Apple so long to get on the LLM train?" folks. Per usual, they lag a bit and then do it better than anyone else.
- JimDabell 2y agoIt’s also because they have a twelve month release cycle.
- gigel82 2y agoThe only way to trust this is them selling "cloud compute" servers that folks can deploy and monitor in their own infrastructure. Nothing else can be guaranteed to not include malicious code to exfiltrate the data. Or better yet, make the APIs public and pluggable so that one can choose an off-device AI processor themselves if one is needed.
- astrange 2y agoYour own infrastructure is definitely less secure than this or even, say, Google. You do not have the capability and teams of SREs to detect intrusions, and an attacker would know that your server processes your data.
- gigel82 2y agoMaybe, but I can totally firewall my own servers to my heart's desire, including completely blocking it off from the internet and only allowing connections via my own network's routes.
- astrange 2y agoSure, but then it doesn't work when you're out of the house, which isn't a good pairing with a phone.
- gigel82 2y agoI VPN (WireGuard) into my home network to access other selfhosted services (like Immich for photos, paperless-ngx, DNS adblock, etc.) and to prevent some tracking, so it would work great for me.
- tiffanyh 2y agoI wonder who Apple will be colocating with for data centers. And what the PCC chassis looks like for these compute devices (will it be a display-less iPad)?
- jrk 2y agoThe have built and operated a growing number of their own data centers for years. Presumably this will go into those.
- jnaina 2y agoStarts with A and ends with S
- tiffanyh 2y agoApple DatacenterS Gotcha, makes sense :)
- jachee 2y agoApple’s rich enough to build and own their own datacenters. Savvy enough, too. I’d imagine the chassis are custom Apple-NOC-specific M-chip powered servers.
- nerdright 2y agoEven if you don't like Apple's monopolistic approaches, you have to admire how they go an extra mile to stay true to their mantra of selling privacy. This is clearly a company with an identity, unlike Microsoft and Google who are very confused.
- vlovich123 2y agoWhat I haven’t heard from the announcement is whether the private cloud has external network access. Presumably it wouldn’t otherwise the guarantees of your request staying in your cloud is meaningless. Conversely, a lot of trivial network stuff can be involved (eg downloading the model). Anyone know which balance Apple is choosing to strike initially?
- CGamesPlay 2y agoAll of this is interesting, but how easy is this to circumvent? When Apple changes their mind for whatever reason, don't they just return a key to a fake PCC node, which would bypass all of their listed protections? Furthermore, what prevents Apple from doing this for specific users?
- a2128 2y agoAccording to the article, it would be difficult to tie any request to a user: > Target diffusion starts with the request metadata, which leaves out any personally identifiable information about the source device or user, and includes only limited contextual data about the request that’s required to enable routing to the appropriate model If this is the case, I wonder how the authentication would work. Is it a security through obscurity sort of situation? Wouldn't it be possible for someone, through extensive reverse engineering, to write a client in Python that gives you a nice free chat API and Apple would be none the wiser?
- filleokus 2y agoDon't know if they use it (or if it would somehow weaken/break the privacy claims you cited), but Apple has an SDK called DeviceCheck[0]. Essentially, your server send a nonce which the client signs using a key pair derived from the Secure Enclave. The server can then verify the signature by an API provided by Apple's servers, and they respond whether it was signed by a Secure Enclave resident key or not. I'm guessing this could be helpful to make it hard(er) to write a Python client. [0]: https://developer.apple.com/documentation/devicecheck/establishing-your-app-s-integrity https://developer.apple.com/documentation/devicecheck/establ...
- JimDabell 2y agoiOS won’t send requests to it unless that node appears in the transparency log. If it appears in the transparency log, the whole world will be able to see that a suspicious node has started serving requests. If Apple changes iOS to remove that restriction, the whole world will be able to see that change because it’s client side. If Apple tries to deliver a custom version of iOS to a single user, the iOS hardware will refuse to run it unless it has a valid signature. If it has a valid signature, that copy of the firmware is irrefutable evidence that Apple is deliberately breaking its privacy promises and spying on people in a way they specifically said they wouldn’t, which would be extremely harmful to their business. Apple seems to be going all-out in binding themselves in a way that makes it as difficult as possible to do what you are suggesting.
- zmmmmm 2y agoRead through it all, it still comes down to "trust us". Apple can sign and authorise an update at any time that will backdoor it, and the government is the stroke of a pen away from forcing them to, all completely silently. I get that there's benefit to what they are doing. But the problem of selling a message of trust is you absolutely have to be 100% truthful about it, and them failing to be transparent that people's data is still subject to access like this poisons the larger message they are selling.
- buzzerbetrayed 2y agoYour argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical. Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well: > Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good. > In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
- devjab 2y agoI think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part: > As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure. Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.
- goupil 2y agoIt's sad to see so many discussions on security and so little on privacy. How about solutions that could combine both, such as homomorphic encryption for AI?
- WatchDog 2y agoI'm interested in how this compares to AWS nitro enclaves, which they mention briefly. The main difference seems to be verifiability down to the firmware level. Nitro enclaves does not provide measurements of the firmware[0], or hypervisor, furthermore they state that the hypervisor code can be updated transparently at any time[1]. Apple is going to provide images of the secure enclave processor operating system(sepOS), as well as the bootloader. It also sounds like they will provide the source code for these components too, although the blog post isn't clear on that. [0]: https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html https://docs.aws.amazon.com/enclaves/latest/user/set-up-atte.... [1]: https://docs.aws.amazon.com/pdfs/whitepapers/latest/security-design-of-aws-nitro-system/security-design-of-aws-nitro-system.pdf https://docs.aws.amazon.com/pdfs/whitepapers/latest/security...
- ram_rattle 2y agoAws had to do it this way because of their custom silicon, Intel, ARM and AMD do provide firmware/hypervisor level attestation
- yolovoe 2y agoNitro does measure firmware. If any firmware is unexpected, server will essentially stop being connected to the EC2 substrate network and/or server wiped clean automatically. People will be paged automatically, security will likely be pulled in, etc. There is no reason to measure hypervisor firmware as it’s not firmware in the case of EC2. The BIOS/UEFI firmware on the mobo is overwritten if it’s tampered with. Hypervisor code (always signed, like all code) is streamed via a verifiably secure system on the server (Nitro cards, which make use of measured boot and/or secure boot). No idea what the customer facing term “Nitro enclaves” means, but EC2 engineers are literally mobilized like an army with pages when any security risk (even minor ones) is determined. Basic stuff like this is covered. We even go as far as guaranteeing core dumps don’t contain any real customer data, even encrypted
- WatchDog 2y agoI'm glad to hear about those internal processes, but I guess the key point of difference is that in apple's case, the measurements of the firmware are provided and verifiable externally. Although in the end, I'm not sure how much of a difference it makes, as ultimately, even with measurements of the whole stack, the platform provider if compelled to do so, can still push out a malicious firmware that fakes it's measurements.
- EternalFury 2y agoLet’s not be too picky. This is a good thing.
- hexage1814 2y agoThe thing with cloud and with anything related to it, anything that connects to the internet somehow... is that, unless it's open source and the servers decentralized, you are always trusting SOMEONE. Sure, Apple might make their best to ensure nobody – but them – have access to your data... but Apple controls all the end points. It controls the updates your iPhone receives, it controls the servers where this happens. Like, they are so many opportunities for them to find what you are doing. It reminds me of this article "Web-based cryptography is always snake oil" https://www.devever.net/~hl/webcrypto https://www.devever.net/~hl/webcrypto And to be fair, this doesn't apply only to this case. Even the data you have stored locally, Apple could access it if they wanted, they sure have power to do it if they so wish or were ordered by the government. They might have done it already and just didn't told anyone for obvious reasons. So, I would argue the best you could say is that it's private in the sense that only Apples knows/can know what you are doing rather than a larger number of entities . Which, you could argue it's a win when the alternatives will leak your data to many more parts... But still far away from being this unbreakable cryptography that it's portrayed it to be.
- seydor 2y ago> if wanted. Or if someone compels them to
- noahtallen 2y agoI don’t think that’s completely fair. It basically puts Apple in the same bucket as Google or OpenAI. Google obviously tracks everything you do for ads, recommendations, AI, you name it. They don’t even hide it, it’s a core part of their business model. Apple, on the other hand, has made a pretty serious effort to ensure that no employee can access your data on these AI systems. That’s hugely different! They’re going as far as to severely restrict logging and observability and even building and designing their own chips and operating systems. And ensuring that clients will refuse to talk to non-audited systems. Yes, we can’t take Apple’s word for it. But I think the third party audits are a huge part of how we trust, and also verify, that this system will be private. I don’t think it’s far to claim that “Apple knows what you’re doing.” That implies that some one, at some level at Apple can at some point access the data sent from your device to this private cloud. That does not seem to be true. I think another facet of trust here is that a rather big part of Apple’s business model is privacy. They’ve been very successful financially by creating products that generate money in other ways, and it’s very much not necessary or even a sound business idea for them to do something else. While I think it’s fair to be skeptical about the claims without 3rd party verification, I don’t think it’s fair to say that Apple’s approach isn’t better for your data and privacy than openAI or Google. (Which I think is the broad implication — openAI tracks prompts for its own model training, not to resell, so it’s also “only openAI knows what your doing.”)
- croes 2y agoWho pays for the costs of private cloud compute, is it free of charge for the iPhone owner (at least until they turn it into a subscription)? What about second hand iPhone users?
- deleted 2y ago[deleted]
- repler 2y agoExactly - nothing is for free. They explicitly state that PCC data gets destroyed after a response is returned. Are the anonymized queries (minus user data context) worth anything? It’s gotta be some kind of subscription/per query charge model to pay for the servers, electricity, and bandwidth.
- AnonHP 2y ago> Who pays for the costs of private cloud compute, is it free of charge for the iPhone owner (at least until they turn it into a subscription)? My guess is that this is similar to how iOS upgrades are “free”, how Apple Maps is free, how iMessage is free, how iCloud Mail is free, etc. To a good extent, it’s all paid for by the price paid by the customer for the hardware. I’d also wager that there will be a paid service/subscription that will get baked into iCloud+ at some point in time (maybe a year from now). This will offer a lot more and Apple will try to attract more customers into its paid services net.
- system7rocks 2y agoI trust Apple
- paul2paul 2y agoWe don't need "a new frontier". I want to be the only one who holds the private key to my encrypted data. I think it's pretty lame to sell privacy when it's not.
- dyauspitr 2y agoThe problem with that is it’s not possible for you to be the only one to hold the private key and have the cloud run your data against a model.
- nardi 2y agoMany people in this thread are extremely cynical and also ignorant of the actual security guarantees. If you don’t think Apple is doing what they say they’re doing, you can go audit the code and prove it doesn’t work. Apple is open sourcing all of it to prove it’s secure and private. If you don’t believe them, the code is right there.
- Marciakhan 2y ago[dead]
- asp_hornet 2y agoThis thread reads like a whole bunch of sour grapes. Hopefully this challenges other companies to do better
- kfreds 2y agoWow! This is incredibly exciting. Apple's Private Cloud Compute seems to be conceptually equivalent with System Transparency - an open-source software project my colleagues and I started six years ago. I'm very much looking forward to more technical details. Should anyone at Apple see this, please feel free to reach out to me at stromberg@mullvad.net. I'd be more than happy to discuss our design, your design, and/or give you feedback. Relevant links: - https://mullvad.net/en/blog/system-transparency-future https://mullvad.net/en/blog/system-transparency-future - http://system-transparency.org http://system-transparency.org (somewhat outdated) - http://sigsum.org http://sigsum.org
- rekoil 2y agoThis was my take from the presentation as well, immediately thought of your feature. Will be interesting to hear your take on it once the details have been made available and fully understood.
- v4dok 2y agohttps://en.m.wikipedia.org/wiki/Confidential_computing https://en.m.wikipedia.org/wiki/Confidential_computing This is what they are doing. Search implementations of this to understand more technical details.
- jiveturkey 2y agoIt's not, AFAICT from the press release. Confidential Compute involves technologies such as SGX and SEV, and for which I think Asylo is an abstraction for (not sure), where the operator (eg Azure) cannot _hardware intercept_ data. The description of what Apple is doing "just" uses their existing code signing and secure boot mechanisms to ensure that everything from the boot firmware (the computers that start before the actual computer starts) to the application, is what you intended it to be. Once it lands in the PCC node it is inspectable though. Confidential Compute goes a step further to ensure that the operator cannot observe the data being operated on, thus also defeating shared workloads that exploit speculative barriers, and hardware bus intercept devices. Confidential Compute also allows attestation of the software being run, something Apple is not providing here. EDIT: looks like they do have attestation, however it's different to how SEV etc attestation works. The client still has to trust that the private key isn't leaked, so this is dependent on other infrastructure working correctly. It also depends on the client getting a correct public key. There's no description of how the client attests that. Interesting that they go through all this effort just for (let's be honest) AI marketing. All your data in the past (location, photos, contacts, safari history) is just as sensitive and deserving of such protection. But apparently PCC will apply only to AI inference workloads. Siri was already and continues to be a kind of cloud AI.
- deleted 2y ago[deleted]
- nisten 2y agoComplete horseshit marketing speak. Was the cloud non-private before? Was it not secure in the first place? Do my Siri searches no longer end up as google ads metadata now? Are the feds no longer able to get rubber stamp access to my i C L O U D now? You are a naive idiot for believing that this is anything but security theater to adress the emotional needs of AI anxiety in and outside the company. Just my opinion.
- v4dok 2y agoThis is Confidential Computing https://en.m.wikipedia.org/wiki/Confidential_computing https://en.m.wikipedia.org/wiki/Confidential_computing with another name. Intel, AMD and Nvidia have been working for years on this. OpenAI released a blog some time ago where they mentioned this as the "next step". Exciting that Apple went ahead and deployed first, it will motivate the rest as well.
- leboshki 2y agoThis new DataProtector tool streamlines confidential computing software development. Imagine being able to rent access to your data so you own it but code running in a TEE can access it, securely transferring ownership of data, or offering subscription bundles for your data. With generative AI essentially turning into an echo chamber where it will train on its own content, sourcing human-derived data and content is going to be so important. This might be how an economy of that data/content gets off the ground. https://medium.com/iex-ec/introducing-the-content-creator-demo-dapp-template-for-developers-c65c03c1bba3 https://medium.com/iex-ec/introducing-the-content-creator-de...
- deleted 2y ago[deleted]
- tzs 2y ago> The Secure Enclave randomizes the data volume’s encryption keys on every reboot and does not persist these random keys, ensuring that data written to the data volume cannot be retained across reboot. In other words, there is an enforceable guarantee that the data volume is cryptographically erased every time the PCC node’s Secure Enclave Processor reboots. I wonder if there is anything that enforces an upper limit on the time between reboots? Since they are building their own chips it would be interesting to include a watchdog timer that runs off an internal oscillator, cannot be disabled by software, and forces a reboot when it expires.
- krosaen 2y agoI wonder if they will ever make this available to developers - I can think of many products that would be nice to have at least part of the cloud infra being hosted in a trusted provider like this, e.g indoor cameras for health metrics: sounds awesome but I would never trust a startup to handle private data this sensitive.
- dymk 2y agoI would love to be able to run a PCC node locally on my M2 MacBook or similar for my iPhone to offload to, even if it’s only for doing what 15 Pro iPhones can do on-device. There’s precedent for this sort of thing as well, like Apple TVs or iPads acting as HomeKit hubs and processing security can footage on-device. Maybe they’ll open that up in the future.
- whatever1 2y agoFyi this is the same company that has been accused of showing people's photos and videos in stranger people's devices by accident. https://discussions.apple.com/thread/252459254?sortBy=best https://discussions.apple.com/thread/252459254?sortBy=best
- blackqueeriroh 2y agoYou mean the _bug_ Apple fixed? These things are not related.
- duskhorizon2 2y ago[dead]
- rldjbpin 2y agonot trusting any of the privacy/security mumbo-jumbo when their icloud free tier still allows for a paltry 5 gigs, when even google offers thrice as much for their public service. i am happy for those who see the positives here, but for the skeptic a toggle to prevent any online processing would be more satisfactory.
- renegade-otter 2y agoI think the best AI business model is charging you to keep the data away from data collection. It's brilliant. Or "Professional" version of software that removes all those annoying "AI" features.
- KETpXDDzR 2y agoThe only way, besides trusting the cloud provider, is encryption. Homomorphic encryption allows you to run calculations on encrypted data without decrypting it. However, besides the performance penalty, it leaks information.