13 ms·

Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Com

by wdewind 13y ago

Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them. Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not. If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one? If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source. The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to). Please correct me where I'm wrong, I'm sure I am!

exelius 13y ago
I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out. What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them. And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
- dllthomas 13y ago
  "Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay." I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
  - exelius 13y ago
    Because the cost of a packet is borne by the receiver while the sender is the one monetizing it. The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
    - wmf 13y ago
      Because the cost of a packet is borne by the receiver... That's true for hot-potato routing, but CDNs generally use cold-potato routing. As for monetization, both Netflix and Comcast are getting paid by their respective customers.
      - exelius 13y ago
        It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute. What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
preinheimer 13y ago
I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers. Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
- exelius 13y ago
  This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
  - mixologic 13y ago
    But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet. I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
    - exelius 13y ago
      But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.
cschroed 13y ago
Comcast promises a certain amount of download capability. When customer requests for content that originate from Comcast's network (that would not normally exceed that download capability) end up saturating a link to which Comcast is party to, then it is Comcast's obligation to attempt to upgrade that link. Comcast cannot control the routes via which the content is returned, they must react to their customers usage patterns to improve the performance of their network. So if tomorrow, everyone wants to download content from some other peer then Comcast will need to upgrade other peer connections. The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion. A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
- mixologic 13y ago
  Not very ballsy if they've bought enough politicians and lobbyists.
  - josephschmoe 13y ago
    He means it's ballsy because any one of us could just start cutting their physical lines.
- jessaustin 13y ago
  ...who's assets are stretched out over thousands of miles and in plain sight... I like the cut of your jib.
johnjac 13y ago
My understanding is this. No Money exchanges hands between Level3 and Comcast, but there is an agreement for Comcast to provide a port for Level3 to connect to. Level3 brings the traffic to Comcast door Step. Level3 pays for the port on their side. Level3 is happy to add more ports, BUT Comcast Refuses. Not because more ports are that expensive but those in charge at Comcast are from the TV side of the house where blackouts are common negotiating tactics in order to be paid. And Level3 refuse to be held over a barrel for payment when Comcast has a local monopoly power.
- deleted 13y ago
  [deleted]
cortesoft 13y ago
I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement. However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network. This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them. Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
- peterwwillis 13y ago
  "Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content." This still misses the mark. Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run. In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry. (Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
zero_intp 13y ago
MSO revenue is threatened by alternate entertainment content, and is terrified of becoming a dumb pipe with no add-on fees. The issue is that Comcast wants to both connect to the internet for free & get paid by both the viewer and the content provider as they are in yesterday's cable model. Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay. Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers. Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps. (veteran employee of a transit network involved)

by wdewind 13y ago

exelius 13y ago

I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out. What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them. And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?

dllthomas 13y ago
"Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay." I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
- exelius 13y ago
  Because the cost of a packet is borne by the receiver while the sender is the one monetizing it. The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
  - wmf 13y ago
    Because the cost of a packet is borne by the receiver... That's true for hot-potato routing, but CDNs generally use cold-potato routing. As for monetization, both Netflix and Comcast are getting paid by their respective customers.
    - exelius 13y ago
      It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute. What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).

preinheimer 13y ago

I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers. Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.

exelius 13y ago
This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
- mixologic 13y ago
  But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet. I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
  - exelius 13y ago
    But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.

cschroed 13y ago

Comcast promises a certain amount of download capability. When customer requests for content that originate from Comcast's network (that would not normally exceed that download capability) end up saturating a link to which Comcast is party to, then it is Comcast's obligation to attempt to upgrade that link. Comcast cannot control the routes via which the content is returned, they must react to their customers usage patterns to improve the performance of their network. So if tomorrow, everyone wants to download content from some other peer then Comcast will need to upgrade other peer connections. The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion. A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)

mixologic 13y ago
Not very ballsy if they've bought enough politicians and lobbyists.
- josephschmoe 13y ago
  He means it's ballsy because any one of us could just start cutting their physical lines.
jessaustin 13y ago
...who's assets are stretched out over thousands of miles and in plain sight... I like the cut of your jib.

johnjac 13y ago

My understanding is this. No Money exchanges hands between Level3 and Comcast, but there is an agreement for Comcast to provide a port for Level3 to connect to. Level3 brings the traffic to Comcast door Step. Level3 pays for the port on their side. Level3 is happy to add more ports, BUT Comcast Refuses. Not because more ports are that expensive but those in charge at Comcast are from the TV side of the house where blackouts are common negotiating tactics in order to be paid. And Level3 refuse to be held over a barrel for payment when Comcast has a local monopoly power.

deleted 13y ago
[deleted]

cortesoft 13y ago

I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement. However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network. This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them. Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.

peterwwillis 13y ago
"Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content." This still misses the mark. Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run. In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry. (Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)

zero_intp 13y ago

MSO revenue is threatened by alternate entertainment content, and is terrified of becoming a dumb pipe with no add-on fees. The issue is that Comcast wants to both connect to the internet for free & get paid by both the viewer and the content provider as they are in yesterday's cable model. Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay. Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers. Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps. (veteran employee of a transit network involved)