10 ms·

Why Your Netflix Traffic Is Slow and Why the FCC Won’t Necessarily Make It Fast

75 points by dajobe 12y ago

GhotiFish 12y ago
Comcast claims that Netflix was sending traffic at such high volumes as to intentionally congest the links between different transit ISPs and Comcast, essentially taking a page from Norton’s “peering playbook” and forcing Comcast and its peers (i.e., the transit providers, Cogent, Level 3, Tata, and others) to upgrade capacity one-by-one, before sending traffic down a different path, congesting that, and forcing an upgrade. >Resolving the dispute: Paid peering (March 2014). Both sides of this argument are reasonable and plausible—this is a classic “peering dispute” Both sides of this argument are reasonable and plausible? Are you kidding me? If Netflix was intentionally shaping their traffic to disrupt Comcast, how is that not a denial of service attack? That sounds like a criminal accusation more than a bargaining position. If Netflix actually was doing that, shouldn't comcast be sueing rather than asking for money for a peering arrangement? also, and this is more to my own ignorance than it is to a counterpoint. I thought peering arrangements were things that happened between ISPs. Netflix is a customer of an ISP.
chrisBob 12y ago
If your ISP wanted to give you good service they could also add a few Netflix OCAs to their rack[1]. Opening up the last mile could allow the competition to use this technique to improve service while using Comcast's (or Verizon's...) network for all of the data. [1]http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450 http://gizmodo.com/this-box-can-hold-an-entire-netflix-15925...
PaulHoule 12y ago
This is why the Net Neutrality ruling was not a big step forward. The right way to solve the internet problem is "unbundling"; that is, the people who own the last mile lease it out to companies that sell you internet, video and other services. If Comcast customers were getting a bad netflix experience they could switch to an ISP that gives a good one.
- k2enemy 12y ago
  How would this solve the problem though? Having competition at the last mile doesn't solve the problem of congestion in miles 2-100 where there is still one company.
  - djrogers 12y ago
    The point is that if 'last-mile' services were open to competition, then there would (could) be many companies in what you call the 2-100 mile arena. Building an ISP infrastructure is merely difficult and expensive. On the other hand, thanks to municipal involvement, lobbying, time-based monopolies, etc. building last-mile everywhere is basically impossible.
    - JoeAltmaier 12y ago
      Exaggeration. Cable did it everywhere in America over a period of a couple of years.
      - dzdt 12y ago
        And cable secured maasive legal and bureaucratic hurdles to prevent anyone else from doing the same.
      - toast0 12y ago
        Cable built an internet infrastructure across America on top of existing outdoor wiring plant (including access to poles) built up over many years. Cable built up the outdoor wiring plant by offering a compelling new service (great reception for local TV and exclusive national programming) without significant competition (you could get a giant satellite dish, but compact dishes were not available until after cable was mostly entrenched). It's not easy to overlay additional infrastructure wiring to compete with existing services (most areas already have good enough internet and video services from one or two providers; especially when a new entrant prompts upgrades from the incumbents). On the other hand, it's very easy to use wholesale access (if available) to provide middle mile competition. You would just need a couple routers (one at last mile provider building; one in a local carrier hotel/colo), a leased line between them, and a transit account.
kokey 12y ago
It's refreshing to see more writing popping up from people that 'get' it. It's a shame that most of it is after people spent a decade campaigning for the wrong thing.
- 01Michael10 12y ago
  Wrong, preserving Net Neutrality was very important but it's just not the whole solution and no one ever said it was... It's called taking the first step. The next step is stopping the TWC/Comcast merger.
pedrocr 12y ago
>Thus, we will certainly see ISPs continue to use prioritization to manage congestion—prioritization is, in fact, a necessary tool for managing limited resources such as bandwidth. Bandwidth may very well be a limited resource in some cases but the dispute between Comcast and Netflix is not one of them. The Comcast network has the bandwidth available to serve the customers (since not all sites are congested) and Netflix is willing to peer for free wherever they want them to, so the slowdown is completely artificial and Comcast generated.
dpark 12y ago
Cogent publicly offered to pay for both sides of the peering upgrades in order to fix their congestion problem. http://www.cogentco.com/en/news/press-releases/631-cogent-offers-to-pay-capital-costs-incurred-by-major-telephone-and-cable-companies-necessary-to-ensure-adequate-capacity http://www.cogentco.com/en/news/press-releases/631-cogent-of... Pretty sure Comcast didn't take them up on the offer. Doing so would have fixed the issue and Netflix wouldn't be writing checks to Comcast now.
- higherpurpose 12y ago
  You mean this check handed right here? http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...
  - msandford 12y ago
    That's Netflix paying Comcast directly, not Cogent giving Comcast the few grand to upgrade the link that they were purposefully not upgrading to force Netflix into paying. EDIT: Also worth noting that Netflix has to pay every month (which is what Comcast wants) rather than Cogent just giving them $10k once for new hardware (not as delicious as lots more money!)
    - ikeboy 12y ago
      It's not just the hardware upgrade, the peering agreement is only free when each side sends the same amount of data. Netflix is sending a lot more than it's receiving, in which case their ISP needs to pay for peering. Edit: I'm just explaining why a hardware upgrade was besides the points that were being made. These are the facts regardless of who you think is in the right. The argument was over bandwidth usage, not hardware upgrades. Is anyone disagreeing with that?
      - msandford 12y ago
        Netflix is only sending the data at the request of Comcast's customers. Comcast is the company which created an asymmetric network such that their customers can request far, far more data than they are able to send. They are reaping the rewards of that decision. It hardly seems reasonable to expect that all the companies that Comcast peers with to have data flowing in equal amounts both ways since the design of Comcast's network explicitly prohibits this. Trying to boil this down to "Netflix is sending a lot more than it's receiving" is to completely ignore WHY this is happening and really makes the debate useless. Symmetric (or nearly symmetric) peering agreements made sense when everyone had symmetric connections like modems, ISDN, T1, T3, etc. When the majority of connections are not symmetric, it makes no sense at all. Again, remember that Netflix isn't DOSing Comcast. Any packets that Netflix sends to Comcast's network was at the request of one of Comcast's (and Netflix's!) customers.
Afforess 12y ago
>In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation. This does not follow. A larger ISP with more market share, regardless of the state of the competition, can extract larger peering agreements. So this merge would absolutely make the current situation for Netflix worse and other peering agreements more expensive.
DannoHung 12y ago
The transit providers have pretty much all agreed with Netflix's side of the story to the best of my knowledge. For example: http://blog.level3.com/open-internet/observations-internet-middleman http://blog.level3.com/open-internet/observations-internet-m...
- ikeboy 12y ago
  That post doesn't mention Netflix at all?
TranceMan 12y ago
Meanwhile the community is trying to create a beneficial framework for all: http://www.open-ix.org/ http://www.open-ix.org/
- wmf 12y ago
  And then evil ISPs will just not join.
icehawk 12y ago
Last I checked it was the consumer ISPs that were stalling with peering upgrades according to multiple transit providers http://blog.level3.com/open-internet/verizons-accidental-mea-culpa/ http://blog.level3.com/open-internet/verizons-accidental-mea... https://news.ycombinator.com/item?id=9264381 https://news.ycombinator.com/item?id=9264381
- ikeboy 12y ago
  Verizon responded to that post. http://publicpolicy.verizon.com/blog/entry/level-3s-selective-amnesia-on-peering http://publicpolicy.verizon.com/blog/entry/level-3s-selectiv...
  - pedrocr 12y ago
    It's a pretty poor response. They're using the fact that Level3 had a dispute with Cogent over unbalanced traffic to say the traffic between Level3 and Verizon should also be balanced. Verizon is not in that condition, it's not a transit provider. Basically there are 4 different situations: 1) ContentProvider<->Level3<->Cogent<->ISP<->User 2) ContentProvider<->Cogent<->Level3<->ISP<->User 3) ContentProvider<->Level3->ISP<->User 4) ContentProvider<->Cogent->ISP<->User In all cases the first network gets a payment by the content provider and the last network a payment by the user. In cases 1 and 2 the middle network is not getting a payment because neither side is their direct client. So the Cogent/Level3 dispute is about there being the same number of 1 and 2 cases so that it all evens out and no compensating payment is necessary. Cases 3 and 4 require no additional payments no matter the upload/download ratio because both network links have been paid for, one by the user the other by the content provider. This whole thing is about the ISPs noticing that the content providers are making money offering content over the internet (shocking I know) and trying to reach across the network and grab some of that revenue, even though everyone has already been compensated for the actual transfer of bits.
    - bcook 12y ago
      >Verizon is not in that condition, it's not a transit provider. Are you sure? http://en.m.wikipedia.org/wiki/Tier_1_network#List_of_tier_1_networks http://en.m.wikipedia.org/wiki/Tier_1_network#List_of_tier_1...
DubiousPusher 12y ago
> And, while it is also worth considering whether the impending merger between Comcast and Time Warner cable might exacerbate the current situation, I predict that this is unlikely. In general, as one access ISP gains more market share, a content provider such as Netflix depends on connectivity to that ISP even more, potentially extracting increasingly higher prices for interconnection. In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation. I'm having trouble with this logic. Increased leverage over content providers seems like it would exactly exacerbate this situation.
ikeboy 12y ago
I tried explaining this to people before the FCC stuff passed, but they just insisted that evil comcast was slowing down netflix streams.
- skylan_q 12y ago
  They want someone to blame for not being able to watch Game of Thrones in HD or for fear of having to pay extra to continue doing so. Providers who fell in between the eyeballs and the content got in the crosshairs. This is really "Game of Thrones" vs. "The Internet" and GoT won.
msandford 12y ago
Yet again, someone fails to understand that this is Comcast abusing their (effective) monopoly to extract additional money via double-paying. First I pay for internet access. 50Mbps "theoretical max" but it's fairly reasonable to expect to get say 10% of that most of the time. Then Netflix has to pay again, in order for me to get the internet traffic that I requested. If Netflix doesn't pay, then Comcast schedules upgrades more slowly than they normally would such that my Netflix experience is degraded. They're not reducing capacity, they're just growing capacity in some areas much, much more slowly than in the rest. To get extra cash.
- toast0 12y ago
  I've never understood the "double pay" argument. You pay for internet. Netflix has to pay for internet too. To deliver packets to Comcast's network, Netflix can chose to pay any number of transit companies, or they can cut out the middle man and pay Comcast. Why is it ok for Netflix to pay a transit company, but not Comcast?
  - pedrocr 12y ago
    > Why is it ok for Netflix to pay a transit company, but not Comcast? Because transit providers are actually providing a service to Netflix. They allow them to dump all the traffic in one location and then the transit provider will build out a network to take care of moving the traffic to anywhere in the globe where there's an ISP requesting it. The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user. When Netflix connects directly to Comcast that's not what's happening, Netflix has to get the content to Comcast's network by itself and the Comcast network is already paid for by the user. So if Netflix pays Comcast then Comcast is getting paid to create the network by both the user and Netflix.
    - jdmichal 12y ago
      > The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user. The "not pay" part is, I assume, based on the prevalent use of settlement-free peering in the industry. These agreements were based on a understanding that, over time, the traffic would wash out even between the two, so let's just not keep track and call it even from the start. This is not necessarily a given in all situations. Comcast asserted that the quantity of Netflix traffic was breaking this understanding.
tzs 12y ago
With Comcast, it's almost obvious that the net neutrality won't change this because Comcast was already operating under net neutrality requirements. To win approval of its NBC merger, Comcast agreed to be bound by net neutrality requirements until 2018. When Verizon won its lawsuit to strike down the FCC's 2010 rules that did not change this. Comcast remained bound by its agreement.
vlan0 12y ago
I have always felt that this issue has been overcomplicated. It seems what this really stems from is the classic ISP business model. From a pure network design standpoint, nearly every network has an uplink that is not large enough to sustain simultaneous high usage from most, or all, of its devices. Your classic endpoint data closet example would be a 48-port switch with 1Gbps access ports, and a 10-20Gbps uplink. This problem seems much worse in the ISP world. Many years ago, ISPs could easily oversell their connection. Realistically only worrying about the small percentage of extreme users downloading high volumes of data via Usenet, BitTorrent, or other file sharing services. Life for an ISP was probably pretty great. Now, with the sheer number of households streaming video, it seems to be apparent that ISPs are drastically overselling their connections. It does not even make logical sense to blame a single service if an ISPs upstream connection is saturated. If the ISP did not oversubscribe their connections at such ridiculous rates, this would not be an issue. Why does it matter that a single service is causing the largest portion of the traffic congestion? Netflix recommends a 5Mbps connection to stream HD video. Most all devices today would stream at HD quality, if possible. How is it that an ISP such as Comcast doesn’t have the uplink capacity to sustain many users at a mere 5Mbps? I find it absurd that multiple ISPs are overselling their connection to the point where a speed of 1/7 of the average connection in the USA is too much of a strain. It really seems that the level of oversubscription is equivalent to having a data closet with a 48-port 100Mbps switch uplinked at 2Gbps . The whole situation just stinks of pure greed.
- jmccree 12y ago
  Just as few as 5 years ago 50-100x over-subscription rates were common for ISPs. For example 100 homes with 10mb connections would be fed by 10-20mbps. The reality was much closer to 48 port 100mbps switches having 100mb uplinks. Netflix (and other streaming services) have been a huge shock to their business models.
mcguire 12y ago
I'm having significant difficulties with this paragraph: "Comcast claims that Netflix was sending traffic at such high volumes as to intentionally congest the links between different transit ISPs and Comcast, essentially taking a page from Norton’s “peering playbook” and forcing Comcast and its peers (i.e., the transit providers, Cogent, Level 3, Tata, and others) to upgrade capacity one-by-one, before sending traffic down a different path, congesting that, and forcing an upgrade. Their position was that Netflix was sending more traffic through these transit providers than the transit providers could handle, and thus that Netflix or their transit providers should pay to connect to Comcast directly. Comcast also implies that certain transit providers such as Cogent are likely the source of congested paths, a claim that has been explored but not yet conclusively proved one way or the other, owing to the difficulty of locating these points of congestion (more on that in a future post)." Now, peering negotiations are an ugly, unpleasant activity that I'm quite happy never to have been a part of. But I don't find this particular claim to be "reasonable and plausible", in the author's words. Where is the congestion? * Inside Netflix? Nope. * Between Netflix and their transit providers? Seems unlikely; Netflix is paying them, after all. Such congestion would effect many of Netflix's customers and be against Netflix's interests. * In the transit providers? Possibly, but again, Netflix is paying them. Further, Netflix connecting directly to Comcast is the first thing Netflix would do, if the transit providers were trying to squeeze more money out of them. * Between the transit providers and Comcast? That seems to me to be the most likely interpretation of that paragraph. But if Netflix is doing this intentionally; say, by directing 90% of their traffic to Comcast customers through Tata, those customers are most likely to be unhappy with Netflix, not Comcast---any traffic from a customer out that doesn't go through Tata is fine, so they wouldn't likely perceive any problems with anything but Netflix. Further, that seems to contradict the statement that "[customers] routinely experienced high latency to many Internet destinations every evening during 'prime time'". And if the congestion is not hitting one specific transit provider, then the idea that it is intentional on Netflix's part is silly and Comcast's peering relationships are simply inadequate for the traffic its customers are using. * Inside Comcast? Well, I could see almost anyone wanting to have an unrelated third-party pay for their infrastructure upgrades, but.... Then, there's this: "The best technical solution (and what ultimately happened) is that Netflix and Comcast should interconnect directly." The best technical solution? The best technical solution? (Here's a hint: a fully-connected network is not a good technical solution.) But the ultimate problem here is: "This is where market leverage comes into play: Because most consumers do not have choice in broadband Internet providers, Comcast arguably (and, empirically speaking, as well) has more market leverage: They can afford to ask Netflix to pay for that direct link—a common Internet business relationship called paid peering—because they have more market power." Comcast has more market power because it is a monopoly and the problem with monopolies is precisely that they use their market power to squeeze money out of counter-parties. As an aside, I find Feamster's comment reply, "What is false in Oliver’s presentation (and many of the claims in the popular press) is his claim about what causes* that slowdown. The cause of the slowdown is congestion, not intentional throttling,*" to be simply disingenuous. What exactly is the difference between intentional throttling and intentionally introducing congestion by not upgrading inadequate peering relationships?

75 points by dajobe 12y ago

Comcast claims that Netflix was sending traffic at such high volumes as to intentionally congest the links between different transit ISPs and Comcast, essentially taking a page from Norton’s “peering playbook” and forcing Comcast and its peers (i.e., the transit providers, Cogent, Level 3, Tata, and others) to upgrade capacity one-by-one, before sending traffic down a different path, congesting that, and forcing an upgrade. >Resolving the dispute: Paid peering (March 2014). Both sides of this argument are reasonable and plausible—this is a classic “peering dispute” Both sides of this argument are reasonable and plausible? Are you kidding me? If Netflix was intentionally shaping their traffic to disrupt Comcast, how is that not a denial of service attack? That sounds like a criminal accusation more than a bargaining position. If Netflix actually was doing that, shouldn't comcast be sueing rather than asking for money for a peering arrangement? also, and this is more to my own ignorance than it is to a counterpoint. I thought peering arrangements were things that happened between ISPs. Netflix is a customer of an ISP.

chrisBob 12y ago

If your ISP wanted to give you good service they could also add a few Netflix OCAs to their rack[1]. Opening up the last mile could allow the competition to use this technique to improve service while using Comcast's (or Verizon's...) network for all of the data. [1]http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450 http://gizmodo.com/this-box-can-hold-an-entire-netflix-15925...

PaulHoule 12y ago

This is why the Net Neutrality ruling was not a big step forward. The right way to solve the internet problem is "unbundling"; that is, the people who own the last mile lease it out to companies that sell you internet, video and other services. If Comcast customers were getting a bad netflix experience they could switch to an ISP that gives a good one.

k2enemy 12y ago
How would this solve the problem though? Having competition at the last mile doesn't solve the problem of congestion in miles 2-100 where there is still one company.
- djrogers 12y ago
  The point is that if 'last-mile' services were open to competition, then there would (could) be many companies in what you call the 2-100 mile arena. Building an ISP infrastructure is merely difficult and expensive. On the other hand, thanks to municipal involvement, lobbying, time-based monopolies, etc. building last-mile everywhere is basically impossible.
  - JoeAltmaier 12y ago
    Exaggeration. Cable did it everywhere in America over a period of a couple of years.
    - dzdt 12y ago
      And cable secured maasive legal and bureaucratic hurdles to prevent anyone else from doing the same.
    - toast0 12y ago
      Cable built an internet infrastructure across America on top of existing outdoor wiring plant (including access to poles) built up over many years. Cable built up the outdoor wiring plant by offering a compelling new service (great reception for local TV and exclusive national programming) without significant competition (you could get a giant satellite dish, but compact dishes were not available until after cable was mostly entrenched). It's not easy to overlay additional infrastructure wiring to compete with existing services (most areas already have good enough internet and video services from one or two providers; especially when a new entrant prompts upgrades from the incumbents). On the other hand, it's very easy to use wholesale access (if available) to provide middle mile competition. You would just need a couple routers (one at last mile provider building; one in a local carrier hotel/colo), a leased line between them, and a transit account.

kokey 12y ago

It's refreshing to see more writing popping up from people that 'get' it. It's a shame that most of it is after people spent a decade campaigning for the wrong thing.

01Michael10 12y ago
Wrong, preserving Net Neutrality was very important but it's just not the whole solution and no one ever said it was... It's called taking the first step. The next step is stopping the TWC/Comcast merger.

pedrocr 12y ago

>Thus, we will certainly see ISPs continue to use prioritization to manage congestion—prioritization is, in fact, a necessary tool for managing limited resources such as bandwidth. Bandwidth may very well be a limited resource in some cases but the dispute between Comcast and Netflix is not one of them. The Comcast network has the bandwidth available to serve the customers (since not all sites are congested) and Netflix is willing to peer for free wherever they want them to, so the slowdown is completely artificial and Comcast generated.

dpark 12y ago

Cogent publicly offered to pay for both sides of the peering upgrades in order to fix their congestion problem. http://www.cogentco.com/en/news/press-releases/631-cogent-offers-to-pay-capital-costs-incurred-by-major-telephone-and-cable-companies-necessary-to-ensure-adequate-capacity http://www.cogentco.com/en/news/press-releases/631-cogent-of... Pretty sure Comcast didn't take them up on the offer. Doing so would have fixed the issue and Netflix wouldn't be writing checks to Comcast now.

higherpurpose 12y ago
You mean this check handed right here? http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...
- msandford 12y ago
  That's Netflix paying Comcast directly, not Cogent giving Comcast the few grand to upgrade the link that they were purposefully not upgrading to force Netflix into paying. EDIT: Also worth noting that Netflix has to pay every month (which is what Comcast wants) rather than Cogent just giving them $10k once for new hardware (not as delicious as lots more money!)
  - ikeboy 12y ago
    It's not just the hardware upgrade, the peering agreement is only free when each side sends the same amount of data. Netflix is sending a lot more than it's receiving, in which case their ISP needs to pay for peering. Edit: I'm just explaining why a hardware upgrade was besides the points that were being made. These are the facts regardless of who you think is in the right. The argument was over bandwidth usage, not hardware upgrades. Is anyone disagreeing with that?
    - msandford 12y ago
      Netflix is only sending the data at the request of Comcast's customers. Comcast is the company which created an asymmetric network such that their customers can request far, far more data than they are able to send. They are reaping the rewards of that decision. It hardly seems reasonable to expect that all the companies that Comcast peers with to have data flowing in equal amounts both ways since the design of Comcast's network explicitly prohibits this. Trying to boil this down to "Netflix is sending a lot more than it's receiving" is to completely ignore WHY this is happening and really makes the debate useless. Symmetric (or nearly symmetric) peering agreements made sense when everyone had symmetric connections like modems, ISDN, T1, T3, etc. When the majority of connections are not symmetric, it makes no sense at all. Again, remember that Netflix isn't DOSing Comcast. Any packets that Netflix sends to Comcast's network was at the request of one of Comcast's (and Netflix's!) customers.

Afforess 12y ago

>In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation. This does not follow. A larger ISP with more market share, regardless of the state of the competition, can extract larger peering agreements. So this merge would absolutely make the current situation for Netflix worse and other peering agreements more expensive.

DannoHung 12y ago

The transit providers have pretty much all agreed with Netflix's side of the story to the best of my knowledge. For example: http://blog.level3.com/open-internet/observations-internet-middleman http://blog.level3.com/open-internet/observations-internet-m...

ikeboy 12y ago
That post doesn't mention Netflix at all?

TranceMan 12y ago

Meanwhile the community is trying to create a beneficial framework for all: http://www.open-ix.org/ http://www.open-ix.org/

wmf 12y ago
And then evil ISPs will just not join.

icehawk 12y ago

Last I checked it was the consumer ISPs that were stalling with peering upgrades according to multiple transit providers http://blog.level3.com/open-internet/verizons-accidental-mea-culpa/ http://blog.level3.com/open-internet/verizons-accidental-mea... https://news.ycombinator.com/item?id=9264381 https://news.ycombinator.com/item?id=9264381

ikeboy 12y ago
Verizon responded to that post. http://publicpolicy.verizon.com/blog/entry/level-3s-selective-amnesia-on-peering http://publicpolicy.verizon.com/blog/entry/level-3s-selectiv...
- pedrocr 12y ago
  It's a pretty poor response. They're using the fact that Level3 had a dispute with Cogent over unbalanced traffic to say the traffic between Level3 and Verizon should also be balanced. Verizon is not in that condition, it's not a transit provider. Basically there are 4 different situations: 1) ContentProvider<->Level3<->Cogent<->ISP<->User 2) ContentProvider<->Cogent<->Level3<->ISP<->User 3) ContentProvider<->Level3->ISP<->User 4) ContentProvider<->Cogent->ISP<->User In all cases the first network gets a payment by the content provider and the last network a payment by the user. In cases 1 and 2 the middle network is not getting a payment because neither side is their direct client. So the Cogent/Level3 dispute is about there being the same number of 1 and 2 cases so that it all evens out and no compensating payment is necessary. Cases 3 and 4 require no additional payments no matter the upload/download ratio because both network links have been paid for, one by the user the other by the content provider. This whole thing is about the ISPs noticing that the content providers are making money offering content over the internet (shocking I know) and trying to reach across the network and grab some of that revenue, even though everyone has already been compensated for the actual transfer of bits.
  - bcook 12y ago
    >Verizon is not in that condition, it's not a transit provider. Are you sure? http://en.m.wikipedia.org/wiki/Tier_1_network#List_of_tier_1_networks http://en.m.wikipedia.org/wiki/Tier_1_network#List_of_tier_1...

DubiousPusher 12y ago

> And, while it is also worth considering whether the impending merger between Comcast and Time Warner cable might exacerbate the current situation, I predict that this is unlikely. In general, as one access ISP gains more market share, a content provider such as Netflix depends on connectivity to that ISP even more, potentially extracting increasingly higher prices for interconnection. In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation. I'm having trouble with this logic. Increased leverage over content providers seems like it would exactly exacerbate this situation.

ikeboy 12y ago

I tried explaining this to people before the FCC stuff passed, but they just insisted that evil comcast was slowing down netflix streams.

skylan_q 12y ago
They want someone to blame for not being able to watch Game of Thrones in HD or for fear of having to pay extra to continue doing so. Providers who fell in between the eyeballs and the content got in the crosshairs. This is really "Game of Thrones" vs. "The Internet" and GoT won.

msandford 12y ago

Yet again, someone fails to understand that this is Comcast abusing their (effective) monopoly to extract additional money via double-paying. First I pay for internet access. 50Mbps "theoretical max" but it's fairly reasonable to expect to get say 10% of that most of the time. Then Netflix has to pay again, in order for me to get the internet traffic that I requested. If Netflix doesn't pay, then Comcast schedules upgrades more slowly than they normally would such that my Netflix experience is degraded. They're not reducing capacity, they're just growing capacity in some areas much, much more slowly than in the rest. To get extra cash.

toast0 12y ago
I've never understood the "double pay" argument. You pay for internet. Netflix has to pay for internet too. To deliver packets to Comcast's network, Netflix can chose to pay any number of transit companies, or they can cut out the middle man and pay Comcast. Why is it ok for Netflix to pay a transit company, but not Comcast?
- pedrocr 12y ago
  > Why is it ok for Netflix to pay a transit company, but not Comcast? Because transit providers are actually providing a service to Netflix. They allow them to dump all the traffic in one location and then the transit provider will build out a network to take care of moving the traffic to anywhere in the globe where there's an ISP requesting it. The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user. When Netflix connects directly to Comcast that's not what's happening, Netflix has to get the content to Comcast's network by itself and the Comcast network is already paid for by the user. So if Netflix pays Comcast then Comcast is getting paid to create the network by both the user and Netflix.
  - jdmichal 12y ago
    > The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user. The "not pay" part is, I assume, based on the prevalent use of settlement-free peering in the industry. These agreements were based on a understanding that, over time, the traffic would wash out even between the two, so let's just not keep track and call it even from the start. This is not necessarily a given in all situations. Comcast asserted that the quantity of Netflix traffic was breaking this understanding.

tzs 12y ago

With Comcast, it's almost obvious that the net neutrality won't change this because Comcast was already operating under net neutrality requirements. To win approval of its NBC merger, Comcast agreed to be bound by net neutrality requirements until 2018. When Verizon won its lawsuit to strike down the FCC's 2010 rules that did not change this. Comcast remained bound by its agreement.

vlan0 12y ago

I have always felt that this issue has been overcomplicated. It seems what this really stems from is the classic ISP business model. From a pure network design standpoint, nearly every network has an uplink that is not large enough to sustain simultaneous high usage from most, or all, of its devices. Your classic endpoint data closet example would be a 48-port switch with 1Gbps access ports, and a 10-20Gbps uplink. This problem seems much worse in the ISP world. Many years ago, ISPs could easily oversell their connection. Realistically only worrying about the small percentage of extreme users downloading high volumes of data via Usenet, BitTorrent, or other file sharing services. Life for an ISP was probably pretty great. Now, with the sheer number of households streaming video, it seems to be apparent that ISPs are drastically overselling their connections. It does not even make logical sense to blame a single service if an ISPs upstream connection is saturated. If the ISP did not oversubscribe their connections at such ridiculous rates, this would not be an issue. Why does it matter that a single service is causing the largest portion of the traffic congestion? Netflix recommends a 5Mbps connection to stream HD video. Most all devices today would stream at HD quality, if possible. How is it that an ISP such as Comcast doesn’t have the uplink capacity to sustain many users at a mere 5Mbps? I find it absurd that multiple ISPs are overselling their connection to the point where a speed of 1/7 of the average connection in the USA is too much of a strain. It really seems that the level of oversubscription is equivalent to having a data closet with a 48-port 100Mbps switch uplinked at 2Gbps . The whole situation just stinks of pure greed.

jmccree 12y ago
Just as few as 5 years ago 50-100x over-subscription rates were common for ISPs. For example 100 homes with 10mb connections would be fed by 10-20mbps. The reality was much closer to 48 port 100mbps switches having 100mb uplinks. Netflix (and other streaming services) have been a huge shock to their business models.

mcguire 12y ago

I'm having significant difficulties with this paragraph: "Comcast claims that Netflix was sending traffic at such high volumes as to intentionally congest the links between different transit ISPs and Comcast, essentially taking a page from Norton’s “peering playbook” and forcing Comcast and its peers (i.e., the transit providers, Cogent, Level 3, Tata, and others) to upgrade capacity one-by-one, before sending traffic down a different path, congesting that, and forcing an upgrade. Their position was that Netflix was sending more traffic through these transit providers than the transit providers could handle, and thus that Netflix or their transit providers should pay to connect to Comcast directly. Comcast also implies that certain transit providers such as Cogent are likely the source of congested paths, a claim that has been explored but not yet conclusively proved one way or the other, owing to the difficulty of locating these points of congestion (more on that in a future post)." Now, peering negotiations are an ugly, unpleasant activity that I'm quite happy never to have been a part of. But I don't find this particular claim to be "reasonable and plausible", in the author's words. Where is the congestion? * Inside Netflix? Nope. * Between Netflix and their transit providers? Seems unlikely; Netflix is paying them, after all. Such congestion would effect many of Netflix's customers and be against Netflix's interests. * In the transit providers? Possibly, but again, Netflix is paying them. Further, Netflix connecting directly to Comcast is the first thing Netflix would do, if the transit providers were trying to squeeze more money out of them. * Between the transit providers and Comcast? That seems to me to be the most likely interpretation of that paragraph. But if Netflix is doing this intentionally; say, by directing 90% of their traffic to Comcast customers through Tata, those customers are most likely to be unhappy with Netflix, not Comcast---any traffic from a customer out that doesn't go through Tata is fine, so they wouldn't likely perceive any problems with anything but Netflix. Further, that seems to contradict the statement that "[customers] routinely experienced high latency to many Internet destinations every evening during 'prime time'". And if the congestion is not hitting one specific transit provider, then the idea that it is intentional on Netflix's part is silly and Comcast's peering relationships are simply inadequate for the traffic its customers are using. * Inside Comcast? Well, I could see almost anyone wanting to have an unrelated third-party pay for their infrastructure upgrades, but.... Then, there's this: "The best technical solution (and what ultimately happened) is that Netflix and Comcast should interconnect directly." The best technical solution? The best technical solution? (Here's a hint: a fully-connected network is not a good technical solution.) But the ultimate problem here is: "This is where market leverage comes into play: Because most consumers do not have choice in broadband Internet providers, Comcast arguably (and, empirically speaking, as well) has more market leverage: They can afford to ask Netflix to pay for that direct link—a common Internet business relationship called paid peering—because they have more market power." Comcast has more market power because it is a monopoly and the problem with monopolies is precisely that they use their market power to squeeze money out of counter-parties. As an aside, I find Feamster's comment reply, "What is false in Oliver’s presentation (and many of the claims in the popular press) is his claim about what causes* that slowdown. The cause of the slowdown is congestion, not intentional throttling,*" to be simply disingenuous. What exactly is the difference between intentional throttling and intentionally introducing congestion by not upgrading inadequate peering relationships?