8 ms·

Those traffic requirements are fairly steep - perhaps it will come down as the program matures. If you're interested in similar edge cache programs: https://o

Those traffic requirements are fairly steep - perhaps it will come down as the program matures. If you're interested in similar edge cache programs: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/ https://www.facebook.com/peering/ https://www.facebook.com/peering/ (though I don't see FNA specifically mentioned there) https://peering.google.com/#/options/google-global-cache https://peering.google.com/#/options/google-global-cache https://www.akamai.com/us/en/products/network-operator/akamai-network-partnerships.jsp https://www.akamai.com/us/en/products/network-operator/akama... https://peering.azurewebsites.net/peering/Caching https://peering.azurewebsites.net/peering/Caching https://www.cloudflare.com/partners/peering-portal/ https://www.cloudflare.com/partners/peering-portal/

ehsankia 7y ago
Do you have a rough overview of how the requirements differ across the similar edge programs linked above? If there's one thing Apple does better, it's providing a clean straight forward page with all the relevant info and little fluff.
- jlgaddis 7y ago
  For reference, Netflix's traffic requirement for OCA is 5 Gbps, IIRC.
- revertts 7y ago
  There are some facility and org requirements depending on the various company's rack setups, but most of that's standard. The main requirement is how much traffic you need to be doing with the peer before they give you a rack - Apple states 25Gbps; more mature programs like Netflix are 5Gbps. Sometimes that's negotiable and they'll hand them out for even smaller traffic amounts (I've heard less than 1Gbps for some of those listed, though won't mention which ones). Like most things in this area of networking, there's a lot of variability and personal contacts involved.
  - toomuchtodo 7y ago
    Some of these deals are even arrived at over beers at a bar at conferences like NANOG (North American Network Operators Group).
    - fyfy18 7y ago
      That's not really suprising. I imagine setting up a new deployment isn't terribly expensive (now the R&D is done), but given you are going to be spending a few weeks working with the ISP to get it up and running, you probably want it to be with someone you get on well with.
      - regularfry 7y ago
        It's not only that. The community is comparatively small and tight-knit, one's technical abilities can be judged relatively comfortably from afar, and social currency matters a lot. Everyone with a stake in The Network already has to trust and be trusted by their peers. In that situation, informal agreements seem to work out well.
        toomuchtodo 7y ago
        Spot on!
- goatsi 7y ago
  Someone on a "start you own ISP" style post a while back mentioned 10Gbps for Google (almost entirely Youtube).
- mike_d 7y ago
  The website isn't really targeted at the people deploying these. If you are pushing more than 10 Gbps into another ASN you'll start getting emails from their peering coordinators. Everyone also meets up twice a year at NANOG.
isclever 7y ago
There is also https://qwilt.com/ https://qwilt.com/ which is more of an open(?) local cache servers for ISPs.
- revertts 7y ago
  Qwilt and similar products are really neat, but I don't think they've entirely panned out. The idea to do transparent caching at the network edge was hurt by the move to TLS everywhere, and there's a lot less cost incentive when things like Open Connect exist (eg. if Netflix will give me a rack that offloads XX% of my total traffic, how much additional traffic do I need this transparent cache to offload before it's cost effective for me?). Without transparent caching the economics get trickier. It seems like their niche is as an easy to deploy, more traditional looking CDN that can run out in the RAN, which does still have value.
teruakohatu 7y ago
I wonder how many peaks Apple TV+ users is required to hit that. The average bitrate is 30mbs so maybe 834 simultaneous 4K streamers.
- yreg 7y ago
  For now, I think the majority of bandwidth would be the zillions of iPhone users doing backups and app updates all the time. However, this is definitely a way to pave the path for Apple TV+.
  - comex 7y ago
    Uploading backups couldn't really be served by a cache. App updates, sure.
    - mike_d 7y ago
      Yes it can. You buffer uploads at an edge server and then drain them at non-peak times. Edge nodes don't have to be just caches. Plenty of major sites do compute too.
      - yzmtf2008 7y ago
        That’s a sure-fire way of getting dataloss. Wouldn’t it be much easier to just upload the backups from users devices when, idk, at night? iCloud requires your device to be locked, connected to WiFi, and charging, so I imagine that already shifts the traffic to off-peak hours.
        ksec 7y ago
        Not sure it is still the case. MacOS Server have this function if you set it up your network, it will cache all your backup, App Update, iCloud Files and Photos on your Mac. Something I wish they do for iOS Time Capsule.
  - divbzero 7y ago
mortdeus 7y ago
dude i didn't even know there was like a whole "developers" universe to netflix.
- sp332 7y ago
  Their most popular thing on HN was probably the Chaos Monkey. https://hn.algolia.com/?query=techblog.netflix.com https://hn.algolia.com/?query=techblog.netflix.com
  - yreg 7y ago
    I really liked their pseudo localization trick. https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe https://netflixtechblog.com/pseudo-localization-netflix-12ff...
    - tialaramex 7y ago
      Simpler tricks but along the same lines have been described on, IIRC The Old New Thing (Raymond Chen's blog for Microsoft) although it's possible it was Michael Kaplan in which case you'd have to dredge the Internet Archive or something because Microsoft deleted Kaplan's blog years ago, then they fired him, and then he died (he had a degenerative disease, although getting fired basically as an exercise in plumping the numbers for the stock market probably isn't good for you either).
alpb 7y ago
Note that peering != edge cache.
- revertts 7y ago
  Yes, but the peering/traffic teams are the ones that generally manage these embedded cache deployments, hence why related info are on the same pages as peering details or in PeeringDB. All the ones I linked do actually have edge cache racks, not just peering arrangements.
grecy 7y ago
A neat secret about these cache devices is ISPs that charge for usage. The data the customers use comes straight from the ISP's Central Office, not from "The internet" at all.
- dgemm 7y ago
  How is that a secret?
  - peterkelly 7y ago
    I didn't know it, and I'm guessing I'm not the only one. https://xkcd.com/1053/ https://xkcd.com/1053/
    - kbutler 7y ago
      Hear, hear - celebrate learning, not membership in the "in-the-know" crowd.
      - makapuf 7y ago
        C'mon, everyone knows that.
    - FortCollinsDev 7y ago
      Thanks for the share!
  - grecy 7y ago
    The ISP I worked for worked very, very hard to keep it a secret from customers, forcing everyone involved to sign an NDA, etc.
    - dgemm 7y ago
      Well, as a subscriber everything beyond your cable modem is "the internet" and you don't care how the ISP's network is set up internally. CDNs, netflix, etc have all been embedding devices in carrier networks for years. Some of your ISP's connectivity probably comes via settlement-free peering, yet we pay for that too. None of this is a secret, it's just how the internet's plumbing works.
RyJones 7y ago
We did something similar at MSN in the 90s - once a dialup ISP was spending so much on transit to us, we’d install a dedicated circuit directly from MSN to whatever mom & pop ISP. It was nice for everyone.
- yrro 7y ago
  Do you know who generally pays who for these services? The ISP saves on transit but racks aren't free either... (I suppose the flippant argument is 'the end user' but that's not quite what I mean...) :)
  - RyJones 7y ago
    MSN paid. We also carried all traffic for Microsoft properties; if you’re an ISP, whatever you were paying for Microsoft data went to zero and your other links had more capacity.
  - revertts 7y ago
    It's a net win for the ISP due to cost savings from not having to haul that traffic across their network/to the Internet. You're right that powering racks isn't free either though: if the cache operator doesn't utilize the rack well and isn't offloading enough traffic to it, the ISP will give them the boot (as in, "we wheeled your rack out to our docking bay, come pick up your piece of junk if you want it back"). It's uncommon since _most_ of the orgs running these racks are competent, but does happen from time to time. Also, even though it's a net win for the ISPs, there are still cases where the operator ends up paying them a fee. This has less to do with the economics of edge caching within an ISP network and more to do with the bargaining power certain ISPs have. The 2014 Netflix/Comcast peering agreement is a good example of how those things sometimes pan out.

ehsankia 7y ago

Do you have a rough overview of how the requirements differ across the similar edge programs linked above? If there's one thing Apple does better, it's providing a clean straight forward page with all the relevant info and little fluff.

jlgaddis 7y ago
For reference, Netflix's traffic requirement for OCA is 5 Gbps, IIRC.
revertts 7y ago
There are some facility and org requirements depending on the various company's rack setups, but most of that's standard. The main requirement is how much traffic you need to be doing with the peer before they give you a rack - Apple states 25Gbps; more mature programs like Netflix are 5Gbps. Sometimes that's negotiable and they'll hand them out for even smaller traffic amounts (I've heard less than 1Gbps for some of those listed, though won't mention which ones). Like most things in this area of networking, there's a lot of variability and personal contacts involved.
- toomuchtodo 7y ago
  Some of these deals are even arrived at over beers at a bar at conferences like NANOG (North American Network Operators Group).
  - fyfy18 7y ago
    That's not really suprising. I imagine setting up a new deployment isn't terribly expensive (now the R&D is done), but given you are going to be spending a few weeks working with the ISP to get it up and running, you probably want it to be with someone you get on well with.
    - regularfry 7y ago
      It's not only that. The community is comparatively small and tight-knit, one's technical abilities can be judged relatively comfortably from afar, and social currency matters a lot. Everyone with a stake in The Network already has to trust and be trusted by their peers. In that situation, informal agreements seem to work out well.
      - toomuchtodo 7y ago
        Spot on!
goatsi 7y ago
Someone on a "start you own ISP" style post a while back mentioned 10Gbps for Google (almost entirely Youtube).
mike_d 7y ago
The website isn't really targeted at the people deploying these. If you are pushing more than 10 Gbps into another ASN you'll start getting emails from their peering coordinators. Everyone also meets up twice a year at NANOG.

isclever 7y ago

There is also https://qwilt.com/ https://qwilt.com/ which is more of an open(?) local cache servers for ISPs.

revertts 7y ago
Qwilt and similar products are really neat, but I don't think they've entirely panned out. The idea to do transparent caching at the network edge was hurt by the move to TLS everywhere, and there's a lot less cost incentive when things like Open Connect exist (eg. if Netflix will give me a rack that offloads XX% of my total traffic, how much additional traffic do I need this transparent cache to offload before it's cost effective for me?). Without transparent caching the economics get trickier. It seems like their niche is as an easy to deploy, more traditional looking CDN that can run out in the RAN, which does still have value.

teruakohatu 7y ago

I wonder how many peaks Apple TV+ users is required to hit that. The average bitrate is 30mbs so maybe 834 simultaneous 4K streamers.

yreg 7y ago
For now, I think the majority of bandwidth would be the zillions of iPhone users doing backups and app updates all the time. However, this is definitely a way to pave the path for Apple TV+.
- comex 7y ago
  Uploading backups couldn't really be served by a cache. App updates, sure.
  - mike_d 7y ago
    Yes it can. You buffer uploads at an edge server and then drain them at non-peak times. Edge nodes don't have to be just caches. Plenty of major sites do compute too.
    - yzmtf2008 7y ago
      That’s a sure-fire way of getting dataloss. Wouldn’t it be much easier to just upload the backups from users devices when, idk, at night? iCloud requires your device to be locked, connected to WiFi, and charging, so I imagine that already shifts the traffic to off-peak hours.
      - ksec 7y ago
        Not sure it is still the case. MacOS Server have this function if you set it up your network, it will cache all your backup, App Update, iCloud Files and Photos on your Mac. Something I wish they do for iOS Time Capsule.
- divbzero 7y ago

mortdeus 7y ago

dude i didn't even know there was like a whole "developers" universe to netflix.

sp332 7y ago
Their most popular thing on HN was probably the Chaos Monkey. https://hn.algolia.com/?query=techblog.netflix.com https://hn.algolia.com/?query=techblog.netflix.com
- yreg 7y ago
  I really liked their pseudo localization trick. https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe https://netflixtechblog.com/pseudo-localization-netflix-12ff...
  - tialaramex 7y ago
    Simpler tricks but along the same lines have been described on, IIRC The Old New Thing (Raymond Chen's blog for Microsoft) although it's possible it was Michael Kaplan in which case you'd have to dredge the Internet Archive or something because Microsoft deleted Kaplan's blog years ago, then they fired him, and then he died (he had a degenerative disease, although getting fired basically as an exercise in plumping the numbers for the stock market probably isn't good for you either).

alpb 7y ago

Note that peering != edge cache.

revertts 7y ago
Yes, but the peering/traffic teams are the ones that generally manage these embedded cache deployments, hence why related info are on the same pages as peering details or in PeeringDB. All the ones I linked do actually have edge cache racks, not just peering arrangements.

grecy 7y ago

A neat secret about these cache devices is ISPs that charge for usage. The data the customers use comes straight from the ISP's Central Office, not from "The internet" at all.

dgemm 7y ago
How is that a secret?
- peterkelly 7y ago
  I didn't know it, and I'm guessing I'm not the only one. https://xkcd.com/1053/ https://xkcd.com/1053/
  - kbutler 7y ago
    Hear, hear - celebrate learning, not membership in the "in-the-know" crowd.
    - makapuf 7y ago
      C'mon, everyone knows that.
  - FortCollinsDev 7y ago
    Thanks for the share!
- grecy 7y ago
  The ISP I worked for worked very, very hard to keep it a secret from customers, forcing everyone involved to sign an NDA, etc.
  - dgemm 7y ago
    Well, as a subscriber everything beyond your cable modem is "the internet" and you don't care how the ISP's network is set up internally. CDNs, netflix, etc have all been embedding devices in carrier networks for years. Some of your ISP's connectivity probably comes via settlement-free peering, yet we pay for that too. None of this is a secret, it's just how the internet's plumbing works.

RyJones 7y ago

We did something similar at MSN in the 90s - once a dialup ISP was spending so much on transit to us, we’d install a dedicated circuit directly from MSN to whatever mom & pop ISP. It was nice for everyone.

yrro 7y ago
Do you know who generally pays who for these services? The ISP saves on transit but racks aren't free either... (I suppose the flippant argument is 'the end user' but that's not quite what I mean...) :)
- RyJones 7y ago
  MSN paid. We also carried all traffic for Microsoft properties; if you’re an ISP, whatever you were paying for Microsoft data went to zero and your other links had more capacity.
- revertts 7y ago
  It's a net win for the ISP due to cost savings from not having to haul that traffic across their network/to the Internet. You're right that powering racks isn't free either though: if the cache operator doesn't utilize the rack well and isn't offloading enough traffic to it, the ISP will give them the boot (as in, "we wheeled your rack out to our docking bay, come pick up your piece of junk if you want it back"). It's uncommon since _most_ of the orgs running these racks are competent, but does happen from time to time. Also, even though it's a net win for the ISPs, there are still cases where the operator ends up paying them a fee. This has less to do with the economics of edge caching within an ISP network and more to do with the bargaining power certain ISPs have. The 2014 Netflix/Comcast peering agreement is a good example of how those things sometimes pan out.