9 ms·

Dumb question -- how does this work with SSL? Is Apple putting their private keys on these boxes and assuming the ISPs can't/won't extract them?

• t3rabytes 7y ago
  Guessing ISPs won't have access to the boxes in a way that'd let them extract them.
• alanfranz 7y ago
  I can't be sure, but: - It's possible to deliver SSL without putting the keys on boxes; of course there's a need for a network connection to the servers holding keys. It's a tech pioneered by cloudflare, can't rememeber the name. - or, there's no SSL for the content, which is signed otherwise (like ubuntu or debian repos).
  • ignoramous 7y ago
    > ...network connection to the servers holding keys. It's a tech pioneered by cloudflare... They announced Keyless SSL way back when: https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-technical-details/ https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-tec... I'm a fan of Facebook's take on this, though: https://engineering.fb.com/security/delegated-credentials/ https://engineering.fb.com/security/delegated-credentials/ Also see, Google's answer to JWTs: Macaroons-- https://hackingdistributed.com/2014/05/16/macaroons-are-better-than-cookies/ https://hackingdistributed.com/2014/05/16/macaroons-are-bett...
• dwild 7y ago
  As long as it's dumb content, it doesn't need SSL signed by Apple. Apple own the intelligent layer, the one that hold the API. Once you query those, it answer with the location and the hash, which allow you to download it from the distributed box and safely verify the content.
  • elithrar 7y ago
    Disagree. - TLS is still important to stop tampering of video content or images, as well as user privacy over what content was specifically viewed. - Some ISPs have (and still do) intercept plaintext video content -> transcode to a much lower bitrate -> cache that for their users. That hurts the content provider, as they lose visibility (logs/metrics), and the user who may suffer a reduced experience that the content provider can’t easily fix. End-to-end TLS solves that.
    • eknkc 7y ago
      Well the API that is behind TLS can provide content verification data (file hashes). Then the client can reject any tampered stuff even without encryption on the transport.
      • elithrar 7y ago
        That still breaks the experience. Now you have tampered content AND broken clients. TLS does wonders for preventing transparent proxies, DPI & ISPs from modifying or tracking what you see.
        • dwild 7y ago
          > That still breaks the experience. Now you have tampered content AND broken clients. You won't have tampered content if the data is rejected, that's absurd. You can do the same with TLS by the way, that's the whole point of TLS, being able to verify (and thus reject or not) data. Sure it break the client, but you can do that in any situation, just have to unplug a wire ;).
    • dwild 7y ago
      My whole point was about using hash to verify the integrity of the data, so no they wouldn't be able to tamper with anything, that's the whole point of it. You bring a good point about user privacy, and for sure a key can help with that, but at the end of the day, there's not much you can do about this once the physical server is somewhere else, TLS or not, you'll need to trust the one that hold physically that server not to void user privacy. I would still suggest a TLS connection for that server, but it would most probably be self signed with a different root certificate to avoid someone else to be able to make others trust him being Apple over something that wouldn't verify the content with hash coming from platform owned by Apple.
• chedabob 7y ago
  Could each box have a short lived leaf certificate signed by an Apple intermediate/root? Also they could just be signing content at the application layer like APT does, and the transport is plain HTTP.
• revertts 7y ago
  It's a very good question! Figuring out security for these remote racks is one of the hardest parts. For some CDNs their larger POPs will be locked cages with security cameras and all, but these small deployments are significantly more exposed. Exactly how SSL is handled (or not) varies by provider, but one thing I'll mention is that these will typically never have a cert so important that it can't be easily revoked, and the most important data will likely not be flowing across them - exposure is very limited. Using video streaming as an example, one option is to only do TCP termination for example.com (or to not even terminate that domain on the local cache, but back at your main datacenter), then use subdomains with individual certs for the local cache (eg. isp1.cache.example.com). In that case, service calls like login, retrieving the manifest, etc. are secured by the certs you're keeping in your primary dc, then the manifest has a set of https://isp1.cache.example.com https://isp1.cache.example.com URLs pointing to the local cache only for video segments. Another tricky aspect is making sure that your main network treats them as untrusted so someone with local access can't use it to get a foothold into the rest of your infra.
  • regecks 7y ago
    >these will typically never have a cert so important that it can't be easily revoked I think this is specifically addressed with the introduction of TLS Delegated Credentials[1]. This allows the CDN edge to use a very short lived credential in the place of the certificate's private key. It's already supported in evergreen browsers and in certificate profiles from commercial CAs like Digicert. 1. https://tools.ietf.org/html/draft-ietf-tls-subcerts-06 https://tools.ietf.org/html/draft-ietf-tls-subcerts-06
    • revertts 7y ago
      Yup! Some of the names on that draft were people who have previously worked on building these sorts of edge racks, so their experience with this infra helped shape the proposal. It'll be great once it's broadly supported, but that's going to take awhile (or depending on your client mix, an eternity).
  • jjeaff 7y ago
    Correct me if I'm wrong, but my understanding is that revoking a certificate is usually not a very effective mitigation for a stolen cert. Since many clients don't check for revocations.
• cmlndz 7y ago
  It (most likely) works like this: the device request the context directly to the edge server (several strategies available to accomplish this: you can ask a master server for the content who issues a redirect to the edge server, Apple could use a single name server and route via DNS, they could use a single name and IP and route via BGP, etc). The edge server, which is the terminating end in the TLS protocol, is aware of the content being requested. Now the edge server can determine if that content is available or not locally and server it back to the user or request it from the origin server first and then back to the user.
  • revertts 7y ago
    The problem isn't with terminating SSL, it's with keeping your keys safe on exposed infrastructure. A single domain name and DNS to route is uncommon because it doesn't give you fine-grained control of load - you need to be mindful of the rack's capacity, and you also need to make sure that most of that ISP's customers go to the rack/people who aren't that ISP's customers don't go to it. Anycasting isn't going to be great for traffic management or long-lived TCP conns, and if you can avoid the complexity of each rack needing a bgp session into the ISP's network you're going to be much better off. Typically this is going to be directly routed to the rack via a unique DNS name after some form of service call.
• deleted 7y ago
  [deleted]
• snowwrestler 7y ago
  Lots of people answering the technical question, but I have to wonder: why do you think ISPs would break into Apple's equipment just to extract private keys securing media files and software downloads? Do you think ISPs are typically in the business of violating the contracts and security of their partners? What would they gain by doing so? Note that I understand the importance of securing private keys in general. I'm just asking about ISPs because you specifically mentioned them.
  • wtracy 7y ago
    Apple has a history of paranoia and unwillingness to trust their own employees, let alone their business partners. Also, there's a surprising number of colo facilities that are pretty lax about who is allowed to come and go.
  • cure 7y ago
    It's not the ISPs themselves, necessarily. You have a box that you own and that sits in someone else's network, where their security policies and practices apply. Those policies and practices are not going to be the same as yours. Perhaps more importantly, other people have physical access to the hardware. That includes employees of the ISP, but perhaps also 3rd parties (think multi-tenant datacenter, for example). That's why you should treat that box as being in a hostile environment.
  • peteretep 7y ago
    > why do you think ISPs would break into Apple's equipment just to extract private keys securing media files and software downloads Being able to sign some things as Apple / MITM some Apple URLs strikes me as interesting enough to get state-security level interest, at which point you fall back to ISPs being made of people, and people being coercible and bribeable.
  • rndgermandude 7y ago
    Imagine you're a state-level intelligence agency, you often can force ISP located/headquartered in your country to obey, and it's probably easier to infiltrate some small-ish out-of-country ISP than an Apple/Google/Amazon data center. It's not necessarily that the ISPs will want to break into equipment, just that they are the weakest link bad actors can use to target Apple/Apple Customers.
• skunkworker 7y ago
  Some older iTunes downloads were encrypted but over plain HTTP and the decryption keys sent over HTTPS.