5 ms·

What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced n

What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network. Then Netflix comes along, and suddenly 35% of Comcast's customer traffic is all going to a handful of Netflix DSNs? Suddenly Comcast has to redo the whole connection. They have to buy top-shelf hardware just for Netflix. I don't know if this has happened, but it's an interesting thought experiment to see how "just one company" could place disproportionate burden/cost on the ISP, as compared to the case where traffic travels to & from many different entities.

josh2600 12y ago
Note: It is definitely not built under that premise, hence why your consumer bandwidth is asymmetrical... The Tier 1 always wants to ask for more bits than a lower tier operator. If a lower tier operator asks for more bits and tilts the traffic balance, things get interesting. Cogent is not a tier 1 but slings 35% of the internet around at will. It does cost a lot of money to route Netflix's bits and the narrative is hugely helpful to Netflix. I'm not saying this is correct, I'm simply trying to point out that Netflix has a ton to gain from carriers upgrading their network connections and routers. Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.
- iron_ball 12y ago
  If Netflix closed its doors today, presumably Amazon Prime and Hulu would pick up the slack. The problem is not Netflix providing content; the problem is users consuming it. Users want better service than the ISPs are inclined to provide.
- sp332 12y ago
  silverstorm didn't say anything about symmetrical. If Comcast assumes that the various interconnect points are relatively even in traffic usage, but then the few that go to Netflix start getting congested (leaving the rest relatively light), they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic. Edit: not sure why I'm getting downvotes since silverstorm clarified exactly what I wrote I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.
  - pdonis 12y ago
    > they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic You're missing a key element of the situation: Comcast made a deal with Netflix precisely to address this problem. Netflix now has direct connections to Comcast's network, so Netflix traffic does not have to go through a third party. In other words, Comcast now directly controls how they connect to Netflix, how many connection points there are, what the average network distance is between the closest Netflix connection point and their customers, etc. So they have a much cheaper way to even out Netflix traffic: open up more connection points. (Which, btw, AFAIK they already did as part of the deal; they certainly did not come out of the deal having "just a few spots" where they are connected to Netflix. Netflix basically is their own CDN; they have servers with their content on them all over the Internet. So Comcast has all kinds of ways to shorten the average network distance between each of their customers and the closest Netflix endpoint.)
- click170 12y ago
  > Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source. Sounds to me like just another way of rephrasing "ISP Customers are demanding huge network upgrades without any additional revenue source." Netflix does not send packets to you unless you request them.
- Silhouette 12y ago
  Netflix is demanding huge network upgrades without any additional revenue source. I think a lot of us would have more sympathy with the ISPs' situation if they hadn't consistently and for a very long time been promoting their services as ever more valuable by offering ever higher bandwidth, yet keeping quiet about little things like massive over-subscription and high contention ratios that meant if all their users actually tried to use the service that was purportedly being offered then no-one would get anywhere near the advertised speeds they were paying for. With the rise of things like streaming video services and Internet telephony, some people are actually trying to pull significant volumes of data down the pipes relative to the advertised size of pipe available. Everyone's known this was coming for a long time, and no-one more so than the ISPs. The way I see it, if the current situation hurts ISPs that failed to invest or set a realistic pricing model, they'll just have to take their medicine and offer a more realistic deal to customers next time (or fail because they can't compete effectively).
icambron 12y ago
Edit: rewritten because I misunderstood your point. Funny, I actually just added an edit to my post above that discusses this. Comcast adds interconnect capacity as needed. So in at least one way, it's actually easier this way, right? Just one peer to make sure they have capacity for. But either way, it's Comcast's job to run a network that's optimized for its customers needs, not some abstract concept of fairness. And in this hypothetical, it's not on Netflix that streaming lots of content is not compatible with how Comcast happened to build its network. Comcast needs to fix its network so that its customers can get their data.
- sliverstorm 12y ago
  I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.
  - deleted 12y ago
    [deleted]
  - icambron 12y ago
    Oops, my bad. Rewrote my response.
  - tacticus 12y ago
    Thankfully netflix expose their stuff via bgp (as well as other nice things) so you can have multiple paths into their network (and out of yours) without any single giant router. allowing you to balance your traffic. You can even ask them for a freeish box to run inside your network to reduce requests to their CDNs
  - pdonis 12y ago
    > just a small handful of nodes But Netflix is not "just a small number of nodes". They're basically their own CDN; they have servers all over the Internet, precisely in order to not have the problem you describe. What's more, Comcast has a deal with Netflix to directly connect Netflix nodes to Comcast's network; in other words, Comcast directly controls how many paths there are for Netflix content to get into its network. If there aren't enough such paths, they can just open up more.
jjoonathan 12y ago
Netflix isn't "doing" anything to Comcast. Comcast's customers are demanding (and paying for) the asymmetrical traffic. If they have to redo the connection, so be it -- they promised their customers connectivity, and if that means installing new hardware to achieve connectivity so be it.
rhizome 12y ago
Asymmetrical peering has been the rule since the 90s. Think of it as fan-in and fan-out, each of which has different dimensions.
saganus 12y ago
But then doesn't that mean that it's also easier to put a cache in front of those overused switches and solve the problem much more easily? I guess the core issue here is that, from a customer's point of view, the ISP is selling me inet access, regardless of where the data actually goes, so it is the ISP's responsibility to keep their end of the bargain (to provide inet access). As a customer I should not care how is their topology planned, what kind of equipment they use, etc. So basically the ISP though that they could offer T amount of bandwidth, advertised it as U (with U > T) because most customer won't use even close to U, and now that more and more users are actually using U amount of bandwidth (or closer to it) the ISP's infrastructure is suffering because they simply where not prepared for it, i.e. they oversold their capacity. OT: By the way, why downvote the parent? I really thought downvoting is for when the post is off-topic or offensive or something... but more and more I see HN modders downvote because they disagree... really? so you make the post go into an ureadable color (grey) just because you disagree? I don't think it's supposed to be that way, is it?
- corobo 12y ago
  Netflix even makes it relatively easy to do this as an ISP too https://www.netflix.com/openconnect https://www.netflix.com/openconnect
- 31reasons 12y ago
  Good point about U > T. It also happens in Gym business where they oversell their gym capacity betting that 80% of their customers will be too lazy to come exercise everyday. Now imagine if I sell a motivation pill that makes you go to gym everyday. It would be absurd if Gym companies send me a cease and desist letter because i am overburdening their facilities because everyone who takes my pill wants to use their gyms everyday.
  - saganus 12y ago
    Actually, now that I think of it, we could probably name a lot of other services that rely on this business model and do a thought experiment like the gym one. An airline for example; now everyone actually caches their flight because Google's self-driving cars always make it on time, and then they demand that Google stops bringing passengers on time? Or a buffet restaurant, since now everyone goes to the gym, they can actually eat a lot more, thus restaurants complain about, either the gym or the pill maker? I don't know, it would be a strange world to live in if this happened in other industries. Maybe prices would rise too much?
irons 12y ago
It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude. And then there's this: http://arstechnica.com/information-technology/2014/04/bittorrent-netflix-should-defeat-isps-by-switching-to-peer-to-peer/ http://arstechnica.com/information-technology/2014/04/bittor... "[W]hen we ask [ISPs] if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence," Netflix CEO Reed Hastings wrote last month. "That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs."
- marknutter 12y ago
  > It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude. That's because around the time Comcast was building up their networks customer demand was wildly asymmetrical. People simply download far more than they upload and Comcast wisely dedicated more of it's bandwidth to downloads.
  - simoncion 12y ago
    I got the data in the following paragraph from a talk on the subject at an Outerz0ne or maybe a Phreaknic years ago: At the time all cable companies were were building up their data network, they were building that network over top of their existing video distribution network. That network was designed [ages prior to this point] to have tiny, tiny, upstream channels for use by system control and maintenance devices, and huge downstream channels to shove 100+ analog TV channels. Because it would have cost a metric shitload to replace all of that hardware, the cable companies were forced to give their data subscribers pretty large downstream links and relatively meager upstream links.[0] Frankly, the nerds of the day (remember, this is the mid-to-late 1990's) would all have been running home servers, and would have been demanding a symmetric connection so that they would have a substantially cheaper alternative to ISDN. (Note that nerds that remember those days consider asymmetric home Internet connections to have been more harmful than the Eternal September.) [0] Note that this property is pretty meaningless in an HFC network where the cable company could run fiber directly to the home, but chooses not to. [See also ATT U-Verse.]
nitrogen 12y ago
What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network. Consumer ISPs have never expected symmetric bandwidth. If they had, they would sell us 50/50 connections instead of 50/10 or 50/2.

josh2600 12y ago

Note: It is definitely not built under that premise, hence why your consumer bandwidth is asymmetrical... The Tier 1 always wants to ask for more bits than a lower tier operator. If a lower tier operator asks for more bits and tilts the traffic balance, things get interesting. Cogent is not a tier 1 but slings 35% of the internet around at will. It does cost a lot of money to route Netflix's bits and the narrative is hugely helpful to Netflix. I'm not saying this is correct, I'm simply trying to point out that Netflix has a ton to gain from carriers upgrading their network connections and routers. Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source.

iron_ball 12y ago
If Netflix closed its doors today, presumably Amazon Prime and Hulu would pick up the slack. The problem is not Netflix providing content; the problem is users consuming it. Users want better service than the ISPs are inclined to provide.
sp332 12y ago
silverstorm didn't say anything about symmetrical. If Comcast assumes that the various interconnect points are relatively even in traffic usage, but then the few that go to Netflix start getting congested (leaving the rest relatively light), they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic. Edit: not sure why I'm getting downvotes since silverstorm clarified exactly what I wrote I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.
- pdonis 12y ago
  > they're going to have to upgrade just a few spots with really expensive equipment just to deal with Netflix traffic You're missing a key element of the situation: Comcast made a deal with Netflix precisely to address this problem. Netflix now has direct connections to Comcast's network, so Netflix traffic does not have to go through a third party. In other words, Comcast now directly controls how they connect to Netflix, how many connection points there are, what the average network distance is between the closest Netflix connection point and their customers, etc. So they have a much cheaper way to even out Netflix traffic: open up more connection points. (Which, btw, AFAIK they already did as part of the deal; they certainly did not come out of the deal having "just a few spots" where they are connected to Netflix. Netflix basically is their own CDN; they have servers with their content on them all over the Internet. So Comcast has all kinds of ways to shorten the average network distance between each of their customers and the closest Netflix endpoint.)
click170 12y ago
> Another way to view this: Netflix is demanding huge network upgrades without any additional revenue source. Sounds to me like just another way of rephrasing "ISP Customers are demanding huge network upgrades without any additional revenue source." Netflix does not send packets to you unless you request them.
Silhouette 12y ago
Netflix is demanding huge network upgrades without any additional revenue source. I think a lot of us would have more sympathy with the ISPs' situation if they hadn't consistently and for a very long time been promoting their services as ever more valuable by offering ever higher bandwidth, yet keeping quiet about little things like massive over-subscription and high contention ratios that meant if all their users actually tried to use the service that was purportedly being offered then no-one would get anywhere near the advertised speeds they were paying for. With the rise of things like streaming video services and Internet telephony, some people are actually trying to pull significant volumes of data down the pipes relative to the advertised size of pipe available. Everyone's known this was coming for a long time, and no-one more so than the ISPs. The way I see it, if the current situation hurts ISPs that failed to invest or set a realistic pricing model, they'll just have to take their medicine and offer a more realistic deal to customers next time (or fail because they can't compete effectively).

icambron 12y ago

Edit: rewritten because I misunderstood your point. Funny, I actually just added an edit to my post above that discusses this. Comcast adds interconnect capacity as needed. So in at least one way, it's actually easier this way, right? Just one peer to make sure they have capacity for. But either way, it's Comcast's job to run a network that's optimized for its customers needs, not some abstract concept of fairness. And in this hypothetical, it's not on Netflix that streaming lots of content is not compatible with how Comcast happened to build its network. Comcast needs to fix its network so that its customers can get their data.

sliverstorm 12y ago
I am not talking about imbalance in U/D, but rather imbalance in destination (or source). If tremendous amounts of traffic is going to or from just a small handful of nodes, you need tremendous routers & pipes going to those nodes.
- deleted 12y ago
  [deleted]
- icambron 12y ago
  Oops, my bad. Rewrote my response.
- tacticus 12y ago
  Thankfully netflix expose their stuff via bgp (as well as other nice things) so you can have multiple paths into their network (and out of yours) without any single giant router. allowing you to balance your traffic. You can even ask them for a freeish box to run inside your network to reduce requests to their CDNs
- pdonis 12y ago
  > just a small handful of nodes But Netflix is not "just a small number of nodes". They're basically their own CDN; they have servers all over the Internet, precisely in order to not have the problem you describe. What's more, Comcast has a deal with Netflix to directly connect Netflix nodes to Comcast's network; in other words, Comcast directly controls how many paths there are for Netflix content to get into its network. If there aren't enough such paths, they can just open up more.

jjoonathan 12y ago

Netflix isn't "doing" anything to Comcast. Comcast's customers are demanding (and paying for) the asymmetrical traffic. If they have to redo the connection, so be it -- they promised their customers connectivity, and if that means installing new hardware to achieve connectivity so be it.

rhizome 12y ago

Asymmetrical peering has been the rule since the 90s. Think of it as fan-in and fan-out, each of which has different dimensions.

saganus 12y ago

But then doesn't that mean that it's also easier to put a cache in front of those overused switches and solve the problem much more easily? I guess the core issue here is that, from a customer's point of view, the ISP is selling me inet access, regardless of where the data actually goes, so it is the ISP's responsibility to keep their end of the bargain (to provide inet access). As a customer I should not care how is their topology planned, what kind of equipment they use, etc. So basically the ISP though that they could offer T amount of bandwidth, advertised it as U (with U > T) because most customer won't use even close to U, and now that more and more users are actually using U amount of bandwidth (or closer to it) the ISP's infrastructure is suffering because they simply where not prepared for it, i.e. they oversold their capacity. OT: By the way, why downvote the parent? I really thought downvoting is for when the post is off-topic or offensive or something... but more and more I see HN modders downvote because they disagree... really? so you make the post go into an ureadable color (grey) just because you disagree? I don't think it's supposed to be that way, is it?

corobo 12y ago
Netflix even makes it relatively easy to do this as an ISP too https://www.netflix.com/openconnect https://www.netflix.com/openconnect
31reasons 12y ago
Good point about U > T. It also happens in Gym business where they oversell their gym capacity betting that 80% of their customers will be too lazy to come exercise everyday. Now imagine if I sell a motivation pill that makes you go to gym everyday. It would be absurd if Gym companies send me a cease and desist letter because i am overburdening their facilities because everyone who takes my pill wants to use their gyms everyday.
- saganus 12y ago
  Actually, now that I think of it, we could probably name a lot of other services that rely on this business model and do a thought experiment like the gym one. An airline for example; now everyone actually caches their flight because Google's self-driving cars always make it on time, and then they demand that Google stops bringing passengers on time? Or a buffet restaurant, since now everyone goes to the gym, they can actually eat a lot more, thus restaurants complain about, either the gym or the pill maker? I don't know, it would be a strange world to live in if this happened in other industries. Maybe prices would rise too much?

irons 12y ago

It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude. And then there's this: http://arstechnica.com/information-technology/2014/04/bittorrent-netflix-should-defeat-isps-by-switching-to-peer-to-peer/ http://arstechnica.com/information-technology/2014/04/bittor... "[W]hen we ask [ISPs] if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence," Netflix CEO Reed Hastings wrote last month. "That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs."

marknutter 12y ago
> It's not credible to suppose that Comcast's network is built around equal exchange of traffic. The broadband connections they provide are wildly asymmetrical, often by an order of magnitude. That's because around the time Comcast was building up their networks customer demand was wildly asymmetrical. People simply download far more than they upload and Comcast wisely dedicated more of it's bandwidth to downloads.
- simoncion 12y ago
  I got the data in the following paragraph from a talk on the subject at an Outerz0ne or maybe a Phreaknic years ago: At the time all cable companies were were building up their data network, they were building that network over top of their existing video distribution network. That network was designed [ages prior to this point] to have tiny, tiny, upstream channels for use by system control and maintenance devices, and huge downstream channels to shove 100+ analog TV channels. Because it would have cost a metric shitload to replace all of that hardware, the cable companies were forced to give their data subscribers pretty large downstream links and relatively meager upstream links.[0] Frankly, the nerds of the day (remember, this is the mid-to-late 1990's) would all have been running home servers, and would have been demanding a symmetric connection so that they would have a substantially cheaper alternative to ISDN. (Note that nerds that remember those days consider asymmetric home Internet connections to have been more harmful than the Eternal September.) [0] Note that this property is pretty meaningless in an HFC network where the cable company could run fiber directly to the home, but chooses not to. [See also ATT U-Verse.]

nitrogen 12y ago

What if Comcast's network is built around the idea that traffic is exchanged relatively equally? They put in gigabit switches everywhere, a nice fair balanced network. Consumer ISPs have never expected symmetric bandwidth. If they had, they would sell us 50/50 connections instead of 50/10 or 50/2.