6 ms·

The problem here is that Comcast and Verizon don't want to pay for the upstream bandwidth to the Netflix Open Connect CDN which means that as Netflix increases

The problem here is that Comcast and Verizon don't want to pay for the upstream bandwidth to the Netflix Open Connect CDN which means that as Netflix increases quality and gains additional subscribers, the time-weighted bitrate (TWBR, the metric they use to compare ISP performance) will continue to plummet. This isn't to say that all Comcast and Verizon customers will feel this pain, some markets have better performance than others (e.g. SF Bay being an example of poor performance). Edit: Other CDNs such as Akamai, Level 3 and Limelight pay to peer with Comcast; whereas Netflix refuses to since Comcast will simply gouge them for the opportunity. Netflix is increasingly moving away from third-party CDNs and onto their Open Connect CDN which will escalate this to a fever pitch.

infecto 13y ago
Thanks for the great explanation. Moved to the Bay Area a year ago and have gotten to the point of debating canceling my Netflix account. The quality is terrible over comcast. No problem with Prime Videos. Next move I will hopefully go for someone else, maybe Monkey Brains.
RKearney 13y ago
It should be noted that having a Netflix OpenConnect Hardware Appliance is 100% free of charge for ISPs. The device itself is free, but I suppose you could factor in the cost to power the 4U server and the rack space used by it but those costs are negligible compared to the additional peering bandwidth used by not having one. From the FAQ[0] >What does the appliance cost my organization? > The appliances (and any necessary replacements) are provided > to participating ISPs free of charge when used within the > terms of the license agreement. [0](PDF) https://secure.netflix.com/us/layout/signup/deviceinfo/OpenConnectDeploymentGuide-v2.4a.pdf https://secure.netflix.com/us/layout/signup/deviceinfo/OpenC...
- aroch 13y ago
  It's also worth noting that a 10ge flatrate port to Cogent isn't going to cost them that much either -- and then they don't have to deal with any additional hardware, just a crossconnect. But for some reason, paying an additional ~$6K a month so that millions of your subscribers have better service is too high a price to pay
  - twoodfin 13y ago
    10 gigabit can't come close to covering the growing demand, no? During primetime hours, Netflix represents some substantial fraction of total U.S. internet traffic. If you're Comcast, running 25% of the U.S. broadband market, you're seeing a big chunk of that traffic. What if Netflix decides to double their max streaming rate? Or quadruple it? We can argue again about what it means that Comcast sells you a 20Mbit-max connection, but as a practical matter, it will cost them a lot more than $6K/month to keep up with Netflix's growth in subscribers, average daily viewership and stream quality.
    - aroch 13y ago
      10Gbps isn't a lot in terms of total capacity or utilization but even a 100 extra kbps of burstable bandwidth/user can greatly improve the experience. It's the difference between buffering every 30sec and watching with only minor stuttering. Also, bandwidth costs only go down/mbps as you get bigger. $6K/month is what you pay if you're a small business; Comcast or Verizon would get high-level pricing (puts their cost somewhere around ~$3.5K/mn). A 50Gbps commit with another 50Gbps at 95th percentile pricing would would cost them fractions of a cent per customer but would make people much happier. Now they'll never to it because they'd rather keep that ~$25K.
      - twoodfin 13y ago
        100Kbps/subscriber is around 200Gbps for Comcast. And that's just 100Kbps! Netflix is going to keep telling their customers they're getting a suboptimal HD experience (and they'll be right!) until the average stream gets somewhere near Blu-ray's ~20Mbps. Maybe half that if you think codecs are going to get better, but by then 4K will be winding down the pike. The point is that for Comcast and Verizon, there's no end in sight: A few more ports connected in a few more data centers isn't going to do more than make a few subscribers slightly happier for a few months.
        aroch 13y ago
        It doesn't have to be dedicated, just bustable to a reasonably large number of subscribers at a time. Netflix will prefretch data and a small amount of burstable bandwidth goes a long way in viewing quality. This isn't about letting everyone stream SuperHD content at the same time, this is about relieving some pressure around the average bitrate (somewhere around 1900Kbps)
- exelius 13y ago
  Other CDNs (Akamai, Level3, etc.) pay Comcast/TimeWarner to host boxes in their datacenters, and to support interconnects to their private CDN networks. If they start giving it to Netflix for free, how do they continue to charge other companies for the same service? The thing to note is that the established precedent is for companies to pay, because they see value in it. Netflix is trying to get something for free that other companies pay for. Oh, but they slapped the word "Open" on the front of it, so it's about free speech! The ISPs don't give a shit about downstream bandwidth consumption because they get paid by the CDNs for that. What they do care about is last-mile bandwidth; i.e. the bandwidth between their datacenter and the customers' homes. It's not unlimited and there are constraints. That problem doesn't get solved (in fact, it gets worse) with OpenConnect.
hayksaakian 13y ago
Could there be an opportunity for small ISPs who peer well with Netflix? Or does the fact that they usually operate using the Big Boys cables remove any advantages this may bring? Instead of Comcast's TV + Phone + Internet, I would be willing to buy Small ISPs NetFlix + AmazonPrime + Internet package if I knew I could get high quality streams without slowdowns.
- acdha 13y ago
  There definitely seems to be – my local ISP runs ads like this: http://rcn.com/j/rcn-supercharges-your-netflix-experience http://rcn.com/j/rcn-supercharges-your-netflix-experience Since I live in a neighborhood where FIOS, Comcast and RCN are all competing this is particularly attractive – we get rock-solid HD quality on Netflix and the Comcast customers are usually complaining about quality. I'm hoping this strategy proves successful…
michaelt 13y ago
How is that consistent with the sudden drop in bandwidth shown in the article's graph http://cdn.arstechnica.net/wp-content/uploads/2014/02/netflix-verizon-fios-speed-640x337.png http://cdn.arstechnica.net/wp-content/uploads/2014/02/netfli... - which is in mbps so if netflix increased their quality, you'd expect it to remain level. Which do you think is more likely, Verizon gained 15% more users after demand was level for months, or Verizon acted in their own self-interest, doing something they have publicly said they want to do?
- twoodfin 13y ago
  How is that consistent with the sudden drop in bandwidth shown in the article's graph ... which is in mbps so if netflix increased their quality, you'd expect it to remain level. To me, that looks an awful lot like our observed Cogent <-> Verizon/Comcast performance starting around October. It also looks like what happens when a network connection becomes saturated: Performance doesn't drop off linearly, it drops off much faster. Netflix and Comcast/Verizon don't have enough bandwidth between them to support subscriber demand and Netflix's high bit rate streams at peak hours. There are three obstacles to improving that: - Somebody's got to pay for bigger interconnect. This is actually the cheapest part, and Netflix would be happy to forgo the interconnect by putting a few boxes loaded with video cache on Comcast's or Verizon's networks. - Somebody's got to pay for the added bandwidth on Comcast and Verizon's networks to push these video packets to subscribers. Well, duh, Comcast and Verizon should, right? Except now they're at the mercy of Netflix, who can keep loading on subscribers and upping the bit rate until a substantial fraction of home broadband subscribers are trying to pull 10Mbit constantly from 7pm-12am EST. Comcast and Verizon are not thrilled to have to pay for this, especially when it's unclear how they benefit. - And of course: Netflix is in direct competition with Verizon and Comcast's own "OTA" TV and video on demand services. Every dollar Comcast and Verizon spend to make Netflix more appealing for their subscribers is a dollar they're spending encouraging those same subscribers to cancel or scale back their expensive TV packages.
  - lsaferite 13y ago
    Minor point, but if I'm paying for 10Mbit, shouldn't I be able to actually USE that fully? Mind you, I'm not paying for a 1Mbit with 10Mbit bursting. I am the revenue source for my ISP and they should have a vested interest in providing me the service level I'm paying for. EDIT: Also, the conflict of interest with an ISP also being a content provider should be legislated away. We have a horrible situation right now and it's only going to get worse.

infecto 13y ago

Thanks for the great explanation. Moved to the Bay Area a year ago and have gotten to the point of debating canceling my Netflix account. The quality is terrible over comcast. No problem with Prime Videos. Next move I will hopefully go for someone else, maybe Monkey Brains.

RKearney 13y ago

It should be noted that having a Netflix OpenConnect Hardware Appliance is 100% free of charge for ISPs. The device itself is free, but I suppose you could factor in the cost to power the 4U server and the rack space used by it but those costs are negligible compared to the additional peering bandwidth used by not having one. From the FAQ[0] >What does the appliance cost my organization? > The appliances (and any necessary replacements) are provided > to participating ISPs free of charge when used within the > terms of the license agreement. [0](PDF) https://secure.netflix.com/us/layout/signup/deviceinfo/OpenConnectDeploymentGuide-v2.4a.pdf https://secure.netflix.com/us/layout/signup/deviceinfo/OpenC...

aroch 13y ago
It's also worth noting that a 10ge flatrate port to Cogent isn't going to cost them that much either -- and then they don't have to deal with any additional hardware, just a crossconnect. But for some reason, paying an additional ~$6K a month so that millions of your subscribers have better service is too high a price to pay
- twoodfin 13y ago
  10 gigabit can't come close to covering the growing demand, no? During primetime hours, Netflix represents some substantial fraction of total U.S. internet traffic. If you're Comcast, running 25% of the U.S. broadband market, you're seeing a big chunk of that traffic. What if Netflix decides to double their max streaming rate? Or quadruple it? We can argue again about what it means that Comcast sells you a 20Mbit-max connection, but as a practical matter, it will cost them a lot more than $6K/month to keep up with Netflix's growth in subscribers, average daily viewership and stream quality.
  - aroch 13y ago
    10Gbps isn't a lot in terms of total capacity or utilization but even a 100 extra kbps of burstable bandwidth/user can greatly improve the experience. It's the difference between buffering every 30sec and watching with only minor stuttering. Also, bandwidth costs only go down/mbps as you get bigger. $6K/month is what you pay if you're a small business; Comcast or Verizon would get high-level pricing (puts their cost somewhere around ~$3.5K/mn). A 50Gbps commit with another 50Gbps at 95th percentile pricing would would cost them fractions of a cent per customer but would make people much happier. Now they'll never to it because they'd rather keep that ~$25K.
    - twoodfin 13y ago
      100Kbps/subscriber is around 200Gbps for Comcast. And that's just 100Kbps! Netflix is going to keep telling their customers they're getting a suboptimal HD experience (and they'll be right!) until the average stream gets somewhere near Blu-ray's ~20Mbps. Maybe half that if you think codecs are going to get better, but by then 4K will be winding down the pike. The point is that for Comcast and Verizon, there's no end in sight: A few more ports connected in a few more data centers isn't going to do more than make a few subscribers slightly happier for a few months.
      - aroch 13y ago
        It doesn't have to be dedicated, just bustable to a reasonably large number of subscribers at a time. Netflix will prefretch data and a small amount of burstable bandwidth goes a long way in viewing quality. This isn't about letting everyone stream SuperHD content at the same time, this is about relieving some pressure around the average bitrate (somewhere around 1900Kbps)
exelius 13y ago
Other CDNs (Akamai, Level3, etc.) pay Comcast/TimeWarner to host boxes in their datacenters, and to support interconnects to their private CDN networks. If they start giving it to Netflix for free, how do they continue to charge other companies for the same service? The thing to note is that the established precedent is for companies to pay, because they see value in it. Netflix is trying to get something for free that other companies pay for. Oh, but they slapped the word "Open" on the front of it, so it's about free speech! The ISPs don't give a shit about downstream bandwidth consumption because they get paid by the CDNs for that. What they do care about is last-mile bandwidth; i.e. the bandwidth between their datacenter and the customers' homes. It's not unlimited and there are constraints. That problem doesn't get solved (in fact, it gets worse) with OpenConnect.

hayksaakian 13y ago

Could there be an opportunity for small ISPs who peer well with Netflix? Or does the fact that they usually operate using the Big Boys cables remove any advantages this may bring? Instead of Comcast's TV + Phone + Internet, I would be willing to buy Small ISPs NetFlix + AmazonPrime + Internet package if I knew I could get high quality streams without slowdowns.

acdha 13y ago
There definitely seems to be – my local ISP runs ads like this: http://rcn.com/j/rcn-supercharges-your-netflix-experience http://rcn.com/j/rcn-supercharges-your-netflix-experience Since I live in a neighborhood where FIOS, Comcast and RCN are all competing this is particularly attractive – we get rock-solid HD quality on Netflix and the Comcast customers are usually complaining about quality. I'm hoping this strategy proves successful…

michaelt 13y ago

How is that consistent with the sudden drop in bandwidth shown in the article's graph http://cdn.arstechnica.net/wp-content/uploads/2014/02/netflix-verizon-fios-speed-640x337.png http://cdn.arstechnica.net/wp-content/uploads/2014/02/netfli... - which is in mbps so if netflix increased their quality, you'd expect it to remain level. Which do you think is more likely, Verizon gained 15% more users after demand was level for months, or Verizon acted in their own self-interest, doing something they have publicly said they want to do?

twoodfin 13y ago
How is that consistent with the sudden drop in bandwidth shown in the article's graph ... which is in mbps so if netflix increased their quality, you'd expect it to remain level. To me, that looks an awful lot like our observed Cogent <-> Verizon/Comcast performance starting around October. It also looks like what happens when a network connection becomes saturated: Performance doesn't drop off linearly, it drops off much faster. Netflix and Comcast/Verizon don't have enough bandwidth between them to support subscriber demand and Netflix's high bit rate streams at peak hours. There are three obstacles to improving that: - Somebody's got to pay for bigger interconnect. This is actually the cheapest part, and Netflix would be happy to forgo the interconnect by putting a few boxes loaded with video cache on Comcast's or Verizon's networks. - Somebody's got to pay for the added bandwidth on Comcast and Verizon's networks to push these video packets to subscribers. Well, duh, Comcast and Verizon should, right? Except now they're at the mercy of Netflix, who can keep loading on subscribers and upping the bit rate until a substantial fraction of home broadband subscribers are trying to pull 10Mbit constantly from 7pm-12am EST. Comcast and Verizon are not thrilled to have to pay for this, especially when it's unclear how they benefit. - And of course: Netflix is in direct competition with Verizon and Comcast's own "OTA" TV and video on demand services. Every dollar Comcast and Verizon spend to make Netflix more appealing for their subscribers is a dollar they're spending encouraging those same subscribers to cancel or scale back their expensive TV packages.
- lsaferite 13y ago
  Minor point, but if I'm paying for 10Mbit, shouldn't I be able to actually USE that fully? Mind you, I'm not paying for a 1Mbit with 10Mbit bursting. I am the revenue source for my ISP and they should have a vested interest in providing me the service level I'm paying for. EDIT: Also, the conflict of interest with an ISP also being a content provider should be legislated away. We have a horrible situation right now and it's only going to get worse.