6 ms·

What’s amazing to me is how much data they’re pumping out — their bandwidth bills must be insane. Or they did some pretty amazing negotiating contracts consider

What’s amazing to me is how much data they’re pumping out — their bandwidth bills must be insane. Or they did some pretty amazing negotiating contracts considering Netflix is like what, $10-15/mo? And there are many who “binge” many shows likely consuming gigabytes per day (in addition to all the other costs, not least of which is actually making the programming)?

dragontamer 5y ago
One of the funny truths about the business of networks, is that the one who "controls" the most bandwidth has the #1 negotiating spot at the table. Take for example, if say Verizon decided to charge more for bandwidth to Netflix... if Netflix said "no" and went with another provider, then Verizon's customer's would suffer from worse access times to Netflix. Verizon has the advantage in that they have a huge customer base that no one wants to piss off Verizon. So it cuts both ways. Bandwidth becomes not a cost at this scale, but instead a moat.
- harikb 5y ago
  My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. ISPs would have been more than happy to show the middle finger to the WFH engineers, but not to the binge-watching masses.
  - elithrar 5y ago
    > My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. Couldn't agree more. We see the opposite when it comes to broadband monopolies: "barely good enough" DSL infrastructure, congested HFC, and adversarial relationships w.r.t. subscriber privacy and experience. When it became worthwhile to invest in not just peering but also last-mile because poor Netflix/YouTube/Disney+/etc performance was a reason for users to churn away, they invested. This isn't to say that this is all "perfect" for consumers either, but this tension has only been good for consumers vs. what we had in the 90's and early-mid 00's.
  - lotsofpulp 5y ago
    The US is still not ready. The vast majority of people have access to only over subscribed coaxial cable internet, with non existent upload allocation.
    - stepbeek 5y ago
      Anecdotally, I'm always very impressed with the connectivity I see in the US versus my home connection in the UK. I'm fairly certain that the last mile from my suburban home to the cabinet is made of corroded aluminium.
      - lotsofpulp 5y ago
        UK is certainly worse. Even in a well populated suburb 10min from Birmingham, I know a family that can only get ADSL or some other service like that at 2Mbps. I have been spoiled with symmetric 1Gbps up and down fiber for a few years, and using the internet is like turning on the electric or gas or water, you do not ever have to think about it.
      - aix1 5y ago
        I live in central London (zone 2) and the fastest wired broadband service available at my house is slower than 20Mbps. (I'm playing with LTE now, but so far it's been a bit of a mixed bag.)
  - oriolid 5y ago
    The binge-watching masses are easy to satisfy. All it takes for the stream to work is average speed of a few megabits per second, but there's so much caching at client end that high latency and few seconds of total blackout every now and then don't really matter.
  - notriddle 5y ago
    Pandemics are temporary. Binge watching will never go away.
- btgeekboy 5y ago
  Not sure if you realize which ISP you picked, but Verizon and Netflix actually had peering disputes in 2014 which gave me quite the headache at my then-current employer.
js4ever 5y ago
Of course they are not paying $120/TB like AWS public pricing I heard they are paying something between $0.20 (eu/us) to $10 (exotic) per TB based on the region of the world where the traffic is coming from
- 101008 5y ago
  Sorry for the naive question, but to offer those prices, there are two options: A) Amazon is losing money to keep Netflix as their client, B) They are doing profit even with $0.20/TB, which means the $120/TB is, at least, 119.80 profitable. Wow.
  - kristjansson 5y ago
    AWS {in,e}gress pricing strategy is not motivated by their cost of provisioning that service. Cloudflare had a good (if self-motivated) analysis of their cost structure discussed on here a while ago https://news.ycombinator.com/item?id=27930151 https://news.ycombinator.com/item?id=27930151
  - manquer 5y ago
    $0.20/ TB costs is not for their agreements with Amazon, bulk of their video traffic is directly peering with ISPs, AWS largely serves for their APIs, and orchestration infrastructure. Amazon and most Cloud providers do overcharge for b/w. You can buy a OVH/Hetzner type box with a guaranteed un-metered 1 Gbps public bandwidth for ~ $120/month easily, which if fully utilized is equivalent 325TB / month or $3-4/TB, completely ignoring the 8/16 core bare metal server and attached storage you also get. This is SMB/ self-service prices, you can get much deals with basic negotiating and getting into a contract with a DC. One thing to remember though not all bandwidth are equal, CSPs like AWS provide a lot of features such as very elastic scale up on-demand, a lot of protection up to L4 and advanced SDN under the hood to make sure your VMs can leverage the b/w, that is computationally expensive and costly.
  - robocat 5y ago
    Netflix has appliances installed at exchange points that caches most of Netflix. Each appliance peers locally and serves the streams locally. The inbound data stream to fill the cache of the appliance is rate limited and time limited - see https://openconnect.zendesk.com/hc/en-us/articles/360035618071 https://openconnect.zendesk.com/hc/en-us/articles/3600356180... The actual inbound data to the appliance will be higher than the fill because not everything is cached. The outbound stream from the appliance serves consumers. In New Zealand for example, Netflix has 40Gbps of connectivity to a peering exchange in Auckland. https://www.peeringdb.com/ix/97 https://www.peeringdb.com/ix/97 So although total Netflix bandwidth to consumers is massive, it has little in common with the bandwidth you pay for at Amazon. Disclaimer: I am not a network engineer.
cavisne 5y ago
Netflix has many peering relationships with isps https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/ The actual bandwidth is significant though, even compared to something like YouTube https://www.smh.com.au/technology/these-graphs-show-the-impact-netflix-is-having-on-the-australian-internet-20150402-1mdc1i.html https://www.smh.com.au/technology/these-graphs-show-the-impa...
- dmead 5y ago
  there's a graph somewhere showing the bitrate for netflix before and after they paid out extortion fees to comcast.
  - NetBeck 5y ago
    Fast.com was originally a tool to measure if your line was throttling Netflix traffic.[0] [0] https://qz.com/688033/netflix-launched-this-handy-speed-test-so-you-can-go-shame-your-internet-provider/ https://qz.com/688033/netflix-launched-this-handy-speed-test...
  - RKearney 5y ago
    Extortion fees? Netflix wanted Comcast to provide them with free bandwidth. While some ISPs may appreciate that arrangement as it offloads the traffic from their transit links, Comcast is under no obligation to do so. You could argue then that Comcast wasn't upgrading their saturated transit links, which they weren't with Level 3, but to assume that every single ISP should provide free pipes to companies is absurd.
    - sedatk 5y ago
      If Netflix pays for the bandwidth, what do Comcast's customers pay for?
      - pstuart 5y ago
        Customer service!
kristjansson 5y ago
Hence, to a large extent, the devices described in the presentation. They’re (co)located with ISP hardware, so the bulk data can be transferred directly to the user at minimal / zero marginal cost
deleted 5y ago
[deleted]

dragontamer 5y ago

One of the funny truths about the business of networks, is that the one who "controls" the most bandwidth has the #1 negotiating spot at the table. Take for example, if say Verizon decided to charge more for bandwidth to Netflix... if Netflix said "no" and went with another provider, then Verizon's customer's would suffer from worse access times to Netflix. Verizon has the advantage in that they have a huge customer base that no one wants to piss off Verizon. So it cuts both ways. Bandwidth becomes not a cost at this scale, but instead a moat.

harikb 5y ago
My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. ISPs would have been more than happy to show the middle finger to the WFH engineers, but not to the binge-watching masses.
- elithrar 5y ago
  > My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. Couldn't agree more. We see the opposite when it comes to broadband monopolies: "barely good enough" DSL infrastructure, congested HFC, and adversarial relationships w.r.t. subscriber privacy and experience. When it became worthwhile to invest in not just peering but also last-mile because poor Netflix/YouTube/Disney+/etc performance was a reason for users to churn away, they invested. This isn't to say that this is all "perfect" for consumers either, but this tension has only been good for consumers vs. what we had in the 90's and early-mid 00's.
- lotsofpulp 5y ago
  The US is still not ready. The vast majority of people have access to only over subscribed coaxial cable internet, with non existent upload allocation.
  - stepbeek 5y ago
    Anecdotally, I'm always very impressed with the connectivity I see in the US versus my home connection in the UK. I'm fairly certain that the last mile from my suburban home to the cabinet is made of corroded aluminium.
    - lotsofpulp 5y ago
      UK is certainly worse. Even in a well populated suburb 10min from Birmingham, I know a family that can only get ADSL or some other service like that at 2Mbps. I have been spoiled with symmetric 1Gbps up and down fiber for a few years, and using the internet is like turning on the electric or gas or water, you do not ever have to think about it.
    - aix1 5y ago
      I live in central London (zone 2) and the fastest wired broadband service available at my house is slower than 20Mbps. (I'm playing with LTE now, but so far it's been a bit of a mixed bag.)
- oriolid 5y ago
  The binge-watching masses are easy to satisfy. All it takes for the stream to work is average speed of a few megabits per second, but there's so much caching at client end that high latency and few seconds of total blackout every now and then don't really matter.
- notriddle 5y ago
  Pandemics are temporary. Binge watching will never go away.
btgeekboy 5y ago
Not sure if you realize which ISP you picked, but Verizon and Netflix actually had peering disputes in 2014 which gave me quite the headache at my then-current employer.

js4ever 5y ago

Of course they are not paying $120/TB like AWS public pricing I heard they are paying something between $0.20 (eu/us) to $10 (exotic) per TB based on the region of the world where the traffic is coming from

101008 5y ago
Sorry for the naive question, but to offer those prices, there are two options: A) Amazon is losing money to keep Netflix as their client, B) They are doing profit even with $0.20/TB, which means the $120/TB is, at least, 119.80 profitable. Wow.
- kristjansson 5y ago
  AWS {in,e}gress pricing strategy is not motivated by their cost of provisioning that service. Cloudflare had a good (if self-motivated) analysis of their cost structure discussed on here a while ago https://news.ycombinator.com/item?id=27930151 https://news.ycombinator.com/item?id=27930151
- manquer 5y ago
  $0.20/ TB costs is not for their agreements with Amazon, bulk of their video traffic is directly peering with ISPs, AWS largely serves for their APIs, and orchestration infrastructure. Amazon and most Cloud providers do overcharge for b/w. You can buy a OVH/Hetzner type box with a guaranteed un-metered 1 Gbps public bandwidth for ~ $120/month easily, which if fully utilized is equivalent 325TB / month or $3-4/TB, completely ignoring the 8/16 core bare metal server and attached storage you also get. This is SMB/ self-service prices, you can get much deals with basic negotiating and getting into a contract with a DC. One thing to remember though not all bandwidth are equal, CSPs like AWS provide a lot of features such as very elastic scale up on-demand, a lot of protection up to L4 and advanced SDN under the hood to make sure your VMs can leverage the b/w, that is computationally expensive and costly.
- robocat 5y ago
  Netflix has appliances installed at exchange points that caches most of Netflix. Each appliance peers locally and serves the streams locally. The inbound data stream to fill the cache of the appliance is rate limited and time limited - see https://openconnect.zendesk.com/hc/en-us/articles/360035618071 https://openconnect.zendesk.com/hc/en-us/articles/3600356180... The actual inbound data to the appliance will be higher than the fill because not everything is cached. The outbound stream from the appliance serves consumers. In New Zealand for example, Netflix has 40Gbps of connectivity to a peering exchange in Auckland. https://www.peeringdb.com/ix/97 https://www.peeringdb.com/ix/97 So although total Netflix bandwidth to consumers is massive, it has little in common with the bandwidth you pay for at Amazon. Disclaimer: I am not a network engineer.

cavisne 5y ago

Netflix has many peering relationships with isps https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/ The actual bandwidth is significant though, even compared to something like YouTube https://www.smh.com.au/technology/these-graphs-show-the-impact-netflix-is-having-on-the-australian-internet-20150402-1mdc1i.html https://www.smh.com.au/technology/these-graphs-show-the-impa...

dmead 5y ago
there's a graph somewhere showing the bitrate for netflix before and after they paid out extortion fees to comcast.
- NetBeck 5y ago
  Fast.com was originally a tool to measure if your line was throttling Netflix traffic.[0] [0] https://qz.com/688033/netflix-launched-this-handy-speed-test-so-you-can-go-shame-your-internet-provider/ https://qz.com/688033/netflix-launched-this-handy-speed-test...
- RKearney 5y ago
  Extortion fees? Netflix wanted Comcast to provide them with free bandwidth. While some ISPs may appreciate that arrangement as it offloads the traffic from their transit links, Comcast is under no obligation to do so. You could argue then that Comcast wasn't upgrading their saturated transit links, which they weren't with Level 3, but to assume that every single ISP should provide free pipes to companies is absurd.
  - sedatk 5y ago
    If Netflix pays for the bandwidth, what do Comcast's customers pay for?
    - pstuart 5y ago
      Customer service!

kristjansson 5y ago

Hence, to a large extent, the devices described in the presentation. They’re (co)located with ISP hardware, so the bulk data can be transferred directly to the user at minimal / zero marginal cost

deleted 5y ago

[deleted]