3 ms·

If this is one server, I can't imagine how much bandwidth Netflix pushes out of a single PoP - does anyone have those numbers, or at least estimates?

wmf 5y ago
I saw a photo somewhere of a Netflix POP that showed dozens of servers and a huge chassis router with hundreds of 100G links, so that's terabits.
- tgtweak 5y ago
  Capacity isn't indicative of bandwidth peak - but I'm sure peaks are well past the terabit measure of each server is being designed for 100-400Gbps.
tgtweak 5y ago
They colocate these servers inside ISPs typically. Would be interesting to know, for the traffic that does go to a Netflix primary pop - how much the hottest one consumes. It feels like a luxury that they can fit their entire content library in all encoded formats into a single server (even though it's a massive server).
- sroussey 5y ago
  It’s not the entire content library. It’s a cache of the most watched at that time.
dylan604 5y ago
There were some crazy number like 80% of internet traffic is Netflix. I have no idea of the validity of that, but without seeing actual numbers, that sounds like a lot.
- kingosticks 5y ago
  There are some numbers at https://www.lightreading.com/carrier-security/security-strategies/cybersecurity-expert-sees-opportunity-in-internet-growth-consolidation/a/d-id/752196 https://www.lightreading.com/carrier-security/security-strat...
zamadatix 5y ago
It may disappoint but the servers are big because it's cheaper than buying racks of servers not because it was the only way to get enough servers into a PoP. Their public IX table should give an idea that most are just a couple https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/ (note they have bulk storage servers too for the less popular content not just 1 big box serving every file). I've seen their boxes walking through Equinix facilities Dallas in Dallas and Chicago and it is a bit jarring how small some of the largest infrastructure can be. It's worth noting that they have many smaller boxes at ISP colo locations as well not just the big regional DCs like Equinix.
- genewitch 5y ago
  Seeing Hulu at equinix, as well as PlayStation Network, in large cages, and then our two small cages was rather eye opening. Some people have a lot of money to throw at a problem, others have smart money to throw at smart engineers.
  - gsnedders 5y ago
    I mean, there's gotta be some question about costs of hardware (plus colocation costs, etc.) versus the costs of engineers to optimise the stack. I don't doubt there's a point at which it's cheaper to focus on reducing hardware and colocation costs, but for the vast majority engineers are the expensive thing.
    - roguas 5y ago
      This is only due to the fact that there is a good chance an engineer on github is already working on your problems and you just wait to integrate his work. Companies that think engineers are expensive will continue to buy tons of hardware and scale rather badly. If you are not actively pushing the ceiling you gonna fall out. You should work on problems cause, who knows, it seems like there might be some value.
  - kristjansson 5y ago
    Someone that worked at one of the former sort of organizations once described it to me as having a 'money hammer'. The money hammer is the solution to all problems. Sometimes, for some organizations, spending 0.1 or 5 or 10 million dollars to solve a problem for now, right now is the smart and prudent choice.

wmf 5y ago

I saw a photo somewhere of a Netflix POP that showed dozens of servers and a huge chassis router with hundreds of 100G links, so that's terabits.

tgtweak 5y ago
Capacity isn't indicative of bandwidth peak - but I'm sure peaks are well past the terabit measure of each server is being designed for 100-400Gbps.

tgtweak 5y ago

They colocate these servers inside ISPs typically. Would be interesting to know, for the traffic that does go to a Netflix primary pop - how much the hottest one consumes. It feels like a luxury that they can fit their entire content library in all encoded formats into a single server (even though it's a massive server).

sroussey 5y ago
It’s not the entire content library. It’s a cache of the most watched at that time.

dylan604 5y ago

There were some crazy number like 80% of internet traffic is Netflix. I have no idea of the validity of that, but without seeing actual numbers, that sounds like a lot.

kingosticks 5y ago
There are some numbers at https://www.lightreading.com/carrier-security/security-strategies/cybersecurity-expert-sees-opportunity-in-internet-growth-consolidation/a/d-id/752196 https://www.lightreading.com/carrier-security/security-strat...

zamadatix 5y ago

It may disappoint but the servers are big because it's cheaper than buying racks of servers not because it was the only way to get enough servers into a PoP. Their public IX table should give an idea that most are just a couple https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/ (note they have bulk storage servers too for the less popular content not just 1 big box serving every file). I've seen their boxes walking through Equinix facilities Dallas in Dallas and Chicago and it is a bit jarring how small some of the largest infrastructure can be. It's worth noting that they have many smaller boxes at ISP colo locations as well not just the big regional DCs like Equinix.

genewitch 5y ago
Seeing Hulu at equinix, as well as PlayStation Network, in large cages, and then our two small cages was rather eye opening. Some people have a lot of money to throw at a problem, others have smart money to throw at smart engineers.
- gsnedders 5y ago
  I mean, there's gotta be some question about costs of hardware (plus colocation costs, etc.) versus the costs of engineers to optimise the stack. I don't doubt there's a point at which it's cheaper to focus on reducing hardware and colocation costs, but for the vast majority engineers are the expensive thing.
  - roguas 5y ago
    This is only due to the fact that there is a good chance an engineer on github is already working on your problems and you just wait to integrate his work. Companies that think engineers are expensive will continue to buy tons of hardware and scale rather badly. If you are not actively pushing the ceiling you gonna fall out. You should work on problems cause, who knows, it seems like there might be some value.
- kristjansson 5y ago
  Someone that worked at one of the former sort of organizations once described it to me as having a 'money hammer'. The money hammer is the solution to all problems. Sometimes, for some organizations, spending 0.1 or 5 or 10 million dollars to solve a problem for now, right now is the smart and prudent choice.