6 ms·

Comcast is definitely throttling Netflix, and it’s infuriating

48 points by ghayes 13y ago

eurleif 13y ago
Comcast is still supposed to be bound by net neutrality rules, unlike other ISPs, because they agreed to follow them until 2018 as part of getting their merger with NBC approved.
jlgaddis 13y ago
FWIW, the author's school has tens of gigabits per seconds (and it may even be 100 Gbps by now) to a facility in which Netflix has a presence. Comcast, AIUI, doesn't.
m_darkTemplar 13y ago
I still see a problem with his testing in that he might be connecting to a server in a different area which is less congested when he connects through his school's VPN. A proper conclusive test would ensure connection to the same Netflix IP somehow over a VPN as his home connection.
- jlgaddis 13y ago
  I live in the same town as the author. Comcast is not my ISP but is the ISP for several of my friends, including friends whom I stayed with for about six weeks recently when I was (for the most part) incapacitated. I can't be 100% certain (I'm going from memory) but I'm almost positive that Netflix traffic did come from the same facility that the author is hitting when connected to the school's VPN. (If it wasn't 2:33 a.m. I'd go verify.) As I mentioned in another comment, however, the .edu is almost certainly better connected to this facility than Comcast is.
eyeareque 13y ago
The author points out that netflix cannot see what is going on over the tunnel: Actually, if Comcast wanted to they could tell what you are doing by using deep packet inspection. Certain firewall vendors have found ways to figure out what you are doing over encrypted tunnels based on protocol pattern/finger printing.
- zwily 13y ago
  They could probably guess he was watching streaming video. It'd be tough for deep packet inspection to tell that it's netflix though.
  - GhotiFish 13y ago
    how could they guess he's streaming a video and not downloading a large file?
    - adrianpike 13y ago
      Timing of packets going both ways. Netflix is a bad example since they use HTTP to wing packets, but with a UDP connection you won't see the regular ACKs heading back upstream.
      - jlgaddis 13y ago
        In this case, the VPN being used is a Juniper SSL VPN, which runs over 443/TCP. Comcast would see TCP ACKs being sent back to the VPN endpoint but they obviously can't see the encrypted data.
jijojv 13y ago
While comcast sucks for its price and monopoly, i think netflix is at fault here. I did a similar test a while back using my company cisco vpn which i personally administer :) and found netflix sucked big time with or without vpn around 9pm any day. Could never get even DVD quality on my 1080p despite being in San Jose, CA with 25Mbps+ crapcast
- bifrost 13y ago
  I wouldn't be so sure, I AnyConnect sucks and Comcast has been heavily congested in SJC for months. I have some graphs I should show you...
wmf 13y ago
This kind of test has been discussed at Ars Technica in the last few days. http://arstechnica.com/information-technology/2014/02/verizon-could-be-throttling-netflix-and-amazon-but-theres-no-actual-evidence-of-it/ http://arstechnica.com/information-technology/2014/02/verizo... It doesn't conclusively prove anything since the Netflix and VPN traffic take different paths, only one of which may be congested. In particular, I suspect the encryption has nothing to do with it. Ultimately I think this is the same old problem: http://www.internap.com/2010/12/02/peering-disputes-comcast-level-3-and-you/ http://www.internap.com/2010/12/02/peering-disputes-comcast-... but as Netflix increases bitrate and number of subscribers it gets worse.
pilif 13y ago
In the last two years, there were periods where YouTube was unusable for me in the early to mid evenings. My provider freely admitted that their current peering arrangement just wasn't good enough for all the traffic that their customers were creating by watching youtube videos. Over time, this was fixed, only to return later, at which point it was fixed again. Huge kudos to my provider (Cablecom in Switzerland) for actually fixing this and even in a somewhat timely manner. But. As more and more of the internet users traffic gets centralized to a few big services (Netflix, YouTube, to some extent Facebook), the usual unbureaucratic peering agreements start to become less and less feasible as traffic starts to be centralized with a few very major content providers. Not only that: some players might start to produce so much traffic that it starts to become useful to go as far as to start doing special proxy solutions for specific content providers (another provider here in Switzerland has a special solution in place for youtube after talking with Google). At which point does all that effort in order to have your end-users be able to access some of these huge content providers stop being worth it in the name of net neutrality? Providing network neutrality is easy when everything just works, but when you have to make new peering agreements, build new pipes or create special infrastructure just for one content provider that doesn't pay you a cent, then that whole neutrality thing becomes very inconvenient and, I have to say, unfair to the providers. Yes. I really, really want network neutrality, but I see that aside of all the political and business-model-preservation issues, we are running into technical issues too and those need solving too.
- izacus 13y ago
  Last I worked at a ISP, Google actually DOES provide YouTube caching servers to ISPs that want it - they bring you a black box machine which you plug into your backbone switch that does transparent caching for users. Getting one has significantly reduced load on internatinal peering lines (I'm from a small EU country) and made quite a differece for our customers as well (suddenly YouTube started working smoothly for most of them while on other ISPs it sometime still stutter). I guess same thing can be provided by Netflix as well.
  - pilif 13y ago
    True. According to RKearney in this thread, Netflix does that too. But isn't that, too, a break of network neutrality? The moment you start accepting content-provider specific equipment, aren't you preferring them over those that don't provide equipment? Wouldn't accepting equipment from one provider compel you to accept from others too in order to stay neutral.
deleted 13y ago
[deleted]
kev009 13y ago
A list of some things that can go wrong: * You could have gotten a lousy route due to poor traffic management or a network issue by your ISP, the transit providers, or a CDN * You could have been given an overloaded server or POP by a CDN's DNS * Your ISP could have a netflix or third party caching appliance, and it could be overloaded * Your ISP could be under-backhauled, under-peered, or under-transited in any points along your route * Your ISP's transit provider(s) could be under-peered, under-backhauled, or under-transited if they are not a tier-1 network. See also poor traffic management and network issues. * Any party along the way to the CDN and to the origin could be under-peered, under-backhauled, or under-transited. See also poor traffic management and network issues. Using a VPN proves very little. Various BGP and DNS tricks are used in the bullets above, and a VPN can drastically change the parameters of all of them.
- tyw 13y ago
  sure, all of that is possible. but honestly how likely is it that some or all of those things are the case on the direct path from netflix/youtube through the ISP in question to so many people's homes? and yet the problem spot is mysteriously avoided when using a VPN (not any particular VPN, but pretty much any of them, judging by most of the people who have spoken out about this problem recently). the problem so effectively avoided by changing the route from the optimal(ish) path to one that is almost guaranteed to be worse from a network flow standpoint. sorry, I don't buy any apologizing for comcast/verizon failing to deliver HD video from netflix/youtube yet somehow able to do it under the same network conditions (same time) just this time through an encrypted VPN. if it was an isolated report... maybe. this is far too widespread to be anything but intentional.
  - kev009 13y ago
    Well, under-provisioned backhaul, peering, and/or transit would cause exactly this. And the blame trail could be quite complex. I'm not apologizing, but commentary by almost anyone other than the network architects and peering coordinators of these organizations are pretty baseless speculations. Disclaimer: I work for a major CDN. Netflix is trying to ween off of us and our competitors, which has contributed to lesser experience. Their model of co-locating gear for free doesn't make much sense to ISPs in the grand scheme of things.
RKearney 13y ago
An alarming number of people don't seem to understand the different between throttling and congestion. Comcast does not participate in Netflix's OpenConnect project. If Comcast wanted to, they could receive a 4U server from Netflix (multiple ones, in fact, for each densely populated area they service) completely free of cost. Obvious costs for power and rack space apply, although the appliance itself is free. This would then allow your Netflix viewing to go to a 4U server at a Comcast datacenter or colocation in the area instead of having the entire South East US try to fit through a small pipe at Marietta, GA. Comcast's stubbornness to participate in the OpenConnect project is the reason for slowness, not throttling. Of course it's going to be faster over a VPN because whatever endpoint you're connecting to most likely takes a different AS path to Netflix.
- Dylan16807 13y ago
  What do you mean 'of course it's going to be faster over a VPN'? The VPN is taking a non-optimal route, it should in almost all cases be of similar or slower speed. If the VPN path is fastest then comcast should be sending data in that direction. Call it congestion if you want but it's intentional congestion caused by intentionally wrong routing.
  - ubercow13 13y ago
    Why do you assume that? Comcast can't just route traffic through IU's private network
    - Dylan16807 13y ago
      If they can't route almost all traffic in the fastest direction, then something is wrong with their network on a technical level. If IU has a massively superior route, then Comcast is negligent to nether peer nor lay their own line along a similar path.
  - RKearney 13y ago
    BGP path selection does not route based on link utilization. What you define as a "non optimal route" isn't what BGP defines as a non optimal route. An ISP can't just send all their customer traffic through any foreign network they choose. The peering location happens to be the "best" path available to Netflix and it won't change until Comcast gets off their high horse and installs Netflix's free caching servers on their network.
bifrost 13y ago
So, let me tell you a story about Comcast and network engineering. Actually, no, I'll skip that. Basically - Comcast runs all of its ports hot. I would say its safe to say what you're seeing is Comcast sucking at paying for upgrades rather than anything specifically malicious.
jlgaddis 13y ago
It's really simple, people. The links that Comcast is carrying Netflix traffic over is getting saturated during peak hours. I am certainly not a fan of Comcast but I don't think they are actively doing anything to screw with Netflix traffic, they just aren't doing anything to fix it.
- roye 13y ago
  This could be tested easily by pinging multiple destinations (besides Netflix) on peak hours
  - jlgaddis 13y ago
    Only if you can ensure that those pings are traversing the same circuits that the Netflix traffic is. (And, even if you can, you can't be sure of the reverse path -- something so many people don't understand when doing traceroutes.)
adventured 13y ago
Being an ISP is not a natural monopoly. This premise is part of the problem, ISPs are being protected from competition when they should not be. From the Wiki entry the author quotes: "A natural monopoly is a monopoly in an industry in which it is most efficient (involving the lowest long-run average cost) for production to be concentrated in a single firm." Competition in the ISP market drives costs lower as it forces the companies to become more efficient, and increases the value proposition offered to customers (greater speeds at an equal or lesser price, thanks Google Fiber!). In fact, I think it would be easy to prove comprehensively that ISPs are in no way natural monopolies. If the author was right, and ISPs were a natural monopoly, then there would be nothing to discuss, the only practical solution would be to install government regulated monopolies in every location across the nation to achieve the absolute lowest cost and consumer price. Google demolished this line of thinking very handily, they showed how instantaneously the government protected telecoms respond to competition in a positive way. The solution is to make it illegal nationally, via Federal law, for any municipality to ever create or encourage an ISP monopoly; and specifically the exact opposite should occur, competition should be heavily encouraged everywhere. The FCC should be mandated with keeping competition lanes wide open in the ISP market, at all times, across the nation. Standards for how cities and towns handle multiple ISPs for infrastructure should be established. Simply: you shall create no law restricting ISP competition or granting monopoly status, period.
- wmf 13y ago
  Competition does encourage efficiency, but competing ISPs are starting with lower efficiency because their infrastructure passes so many homes that aren't connected. Pretty much no one understands how Google Fiber can be profitable at its prices, so unless they open their books or someone replicates that business model I'm not going to call it a solution.
- kalleboo 13y ago
  The OP may be using it wrong, but I still think at this point, the last mile network is a natural monopoly. Where I live, city-owned power companies have built single fiber-to-the-building networks that are the open for ISP competition inside the network (I.e. the ISP you pick just delivers the bandwidth/peering). In my view this is the optimal compromise. This wasn't true just a few years ago when DSL and Cable tech was still competitive, but at this point it seems pretty clear that fiber infrastructure isn't going to be beat.
jlgaddis 13y ago
One other note: the author's school (the entity he brought up the VPN session to) manages many of the largest R&D networks in the country. As such, it is extremely well-connected to, well, just about everything (including Netflix's delivery networks).

48 points by ghayes 13y ago

Comcast is still supposed to be bound by net neutrality rules, unlike other ISPs, because they agreed to follow them until 2018 as part of getting their merger with NBC approved.

jlgaddis 13y ago

FWIW, the author's school has tens of gigabits per seconds (and it may even be 100 Gbps by now) to a facility in which Netflix has a presence. Comcast, AIUI, doesn't.

m_darkTemplar 13y ago

I still see a problem with his testing in that he might be connecting to a server in a different area which is less congested when he connects through his school's VPN. A proper conclusive test would ensure connection to the same Netflix IP somehow over a VPN as his home connection.

jlgaddis 13y ago
I live in the same town as the author. Comcast is not my ISP but is the ISP for several of my friends, including friends whom I stayed with for about six weeks recently when I was (for the most part) incapacitated. I can't be 100% certain (I'm going from memory) but I'm almost positive that Netflix traffic did come from the same facility that the author is hitting when connected to the school's VPN. (If it wasn't 2:33 a.m. I'd go verify.) As I mentioned in another comment, however, the .edu is almost certainly better connected to this facility than Comcast is.

eyeareque 13y ago

The author points out that netflix cannot see what is going on over the tunnel: Actually, if Comcast wanted to they could tell what you are doing by using deep packet inspection. Certain firewall vendors have found ways to figure out what you are doing over encrypted tunnels based on protocol pattern/finger printing.

zwily 13y ago
They could probably guess he was watching streaming video. It'd be tough for deep packet inspection to tell that it's netflix though.
- GhotiFish 13y ago
  how could they guess he's streaming a video and not downloading a large file?
  - adrianpike 13y ago
    Timing of packets going both ways. Netflix is a bad example since they use HTTP to wing packets, but with a UDP connection you won't see the regular ACKs heading back upstream.
    - jlgaddis 13y ago
      In this case, the VPN being used is a Juniper SSL VPN, which runs over 443/TCP. Comcast would see TCP ACKs being sent back to the VPN endpoint but they obviously can't see the encrypted data.

jijojv 13y ago

While comcast sucks for its price and monopoly, i think netflix is at fault here. I did a similar test a while back using my company cisco vpn which i personally administer :) and found netflix sucked big time with or without vpn around 9pm any day. Could never get even DVD quality on my 1080p despite being in San Jose, CA with 25Mbps+ crapcast

bifrost 13y ago
I wouldn't be so sure, I AnyConnect sucks and Comcast has been heavily congested in SJC for months. I have some graphs I should show you...

wmf 13y ago

This kind of test has been discussed at Ars Technica in the last few days. http://arstechnica.com/information-technology/2014/02/verizon-could-be-throttling-netflix-and-amazon-but-theres-no-actual-evidence-of-it/ http://arstechnica.com/information-technology/2014/02/verizo... It doesn't conclusively prove anything since the Netflix and VPN traffic take different paths, only one of which may be congested. In particular, I suspect the encryption has nothing to do with it. Ultimately I think this is the same old problem: http://www.internap.com/2010/12/02/peering-disputes-comcast-level-3-and-you/ http://www.internap.com/2010/12/02/peering-disputes-comcast-... but as Netflix increases bitrate and number of subscribers it gets worse.

pilif 13y ago

In the last two years, there were periods where YouTube was unusable for me in the early to mid evenings. My provider freely admitted that their current peering arrangement just wasn't good enough for all the traffic that their customers were creating by watching youtube videos. Over time, this was fixed, only to return later, at which point it was fixed again. Huge kudos to my provider (Cablecom in Switzerland) for actually fixing this and even in a somewhat timely manner. But. As more and more of the internet users traffic gets centralized to a few big services (Netflix, YouTube, to some extent Facebook), the usual unbureaucratic peering agreements start to become less and less feasible as traffic starts to be centralized with a few very major content providers. Not only that: some players might start to produce so much traffic that it starts to become useful to go as far as to start doing special proxy solutions for specific content providers (another provider here in Switzerland has a special solution in place for youtube after talking with Google). At which point does all that effort in order to have your end-users be able to access some of these huge content providers stop being worth it in the name of net neutrality? Providing network neutrality is easy when everything just works, but when you have to make new peering agreements, build new pipes or create special infrastructure just for one content provider that doesn't pay you a cent, then that whole neutrality thing becomes very inconvenient and, I have to say, unfair to the providers. Yes. I really, really want network neutrality, but I see that aside of all the political and business-model-preservation issues, we are running into technical issues too and those need solving too.

izacus 13y ago
Last I worked at a ISP, Google actually DOES provide YouTube caching servers to ISPs that want it - they bring you a black box machine which you plug into your backbone switch that does transparent caching for users. Getting one has significantly reduced load on internatinal peering lines (I'm from a small EU country) and made quite a differece for our customers as well (suddenly YouTube started working smoothly for most of them while on other ISPs it sometime still stutter). I guess same thing can be provided by Netflix as well.
- pilif 13y ago
  True. According to RKearney in this thread, Netflix does that too. But isn't that, too, a break of network neutrality? The moment you start accepting content-provider specific equipment, aren't you preferring them over those that don't provide equipment? Wouldn't accepting equipment from one provider compel you to accept from others too in order to stay neutral.

deleted 13y ago

[deleted]

kev009 13y ago

A list of some things that can go wrong: * You could have gotten a lousy route due to poor traffic management or a network issue by your ISP, the transit providers, or a CDN * You could have been given an overloaded server or POP by a CDN's DNS * Your ISP could have a netflix or third party caching appliance, and it could be overloaded * Your ISP could be under-backhauled, under-peered, or under-transited in any points along your route * Your ISP's transit provider(s) could be under-peered, under-backhauled, or under-transited if they are not a tier-1 network. See also poor traffic management and network issues. * Any party along the way to the CDN and to the origin could be under-peered, under-backhauled, or under-transited. See also poor traffic management and network issues. Using a VPN proves very little. Various BGP and DNS tricks are used in the bullets above, and a VPN can drastically change the parameters of all of them.

tyw 13y ago
sure, all of that is possible. but honestly how likely is it that some or all of those things are the case on the direct path from netflix/youtube through the ISP in question to so many people's homes? and yet the problem spot is mysteriously avoided when using a VPN (not any particular VPN, but pretty much any of them, judging by most of the people who have spoken out about this problem recently). the problem so effectively avoided by changing the route from the optimal(ish) path to one that is almost guaranteed to be worse from a network flow standpoint. sorry, I don't buy any apologizing for comcast/verizon failing to deliver HD video from netflix/youtube yet somehow able to do it under the same network conditions (same time) just this time through an encrypted VPN. if it was an isolated report... maybe. this is far too widespread to be anything but intentional.
- kev009 13y ago
  Well, under-provisioned backhaul, peering, and/or transit would cause exactly this. And the blame trail could be quite complex. I'm not apologizing, but commentary by almost anyone other than the network architects and peering coordinators of these organizations are pretty baseless speculations. Disclaimer: I work for a major CDN. Netflix is trying to ween off of us and our competitors, which has contributed to lesser experience. Their model of co-locating gear for free doesn't make much sense to ISPs in the grand scheme of things.

RKearney 13y ago

An alarming number of people don't seem to understand the different between throttling and congestion. Comcast does not participate in Netflix's OpenConnect project. If Comcast wanted to, they could receive a 4U server from Netflix (multiple ones, in fact, for each densely populated area they service) completely free of cost. Obvious costs for power and rack space apply, although the appliance itself is free. This would then allow your Netflix viewing to go to a 4U server at a Comcast datacenter or colocation in the area instead of having the entire South East US try to fit through a small pipe at Marietta, GA. Comcast's stubbornness to participate in the OpenConnect project is the reason for slowness, not throttling. Of course it's going to be faster over a VPN because whatever endpoint you're connecting to most likely takes a different AS path to Netflix.

Dylan16807 13y ago
What do you mean 'of course it's going to be faster over a VPN'? The VPN is taking a non-optimal route, it should in almost all cases be of similar or slower speed. If the VPN path is fastest then comcast should be sending data in that direction. Call it congestion if you want but it's intentional congestion caused by intentionally wrong routing.
- ubercow13 13y ago
  Why do you assume that? Comcast can't just route traffic through IU's private network
  - Dylan16807 13y ago
    If they can't route almost all traffic in the fastest direction, then something is wrong with their network on a technical level. If IU has a massively superior route, then Comcast is negligent to nether peer nor lay their own line along a similar path.
- RKearney 13y ago
  BGP path selection does not route based on link utilization. What you define as a "non optimal route" isn't what BGP defines as a non optimal route. An ISP can't just send all their customer traffic through any foreign network they choose. The peering location happens to be the "best" path available to Netflix and it won't change until Comcast gets off their high horse and installs Netflix's free caching servers on their network.

bifrost 13y ago

So, let me tell you a story about Comcast and network engineering. Actually, no, I'll skip that. Basically - Comcast runs all of its ports hot. I would say its safe to say what you're seeing is Comcast sucking at paying for upgrades rather than anything specifically malicious.

jlgaddis 13y ago

It's really simple, people. The links that Comcast is carrying Netflix traffic over is getting saturated during peak hours. I am certainly not a fan of Comcast but I don't think they are actively doing anything to screw with Netflix traffic, they just aren't doing anything to fix it.

roye 13y ago
This could be tested easily by pinging multiple destinations (besides Netflix) on peak hours
- jlgaddis 13y ago
  Only if you can ensure that those pings are traversing the same circuits that the Netflix traffic is. (And, even if you can, you can't be sure of the reverse path -- something so many people don't understand when doing traceroutes.)

adventured 13y ago

Being an ISP is not a natural monopoly. This premise is part of the problem, ISPs are being protected from competition when they should not be. From the Wiki entry the author quotes: "A natural monopoly is a monopoly in an industry in which it is most efficient (involving the lowest long-run average cost) for production to be concentrated in a single firm." Competition in the ISP market drives costs lower as it forces the companies to become more efficient, and increases the value proposition offered to customers (greater speeds at an equal or lesser price, thanks Google Fiber!). In fact, I think it would be easy to prove comprehensively that ISPs are in no way natural monopolies. If the author was right, and ISPs were a natural monopoly, then there would be nothing to discuss, the only practical solution would be to install government regulated monopolies in every location across the nation to achieve the absolute lowest cost and consumer price. Google demolished this line of thinking very handily, they showed how instantaneously the government protected telecoms respond to competition in a positive way. The solution is to make it illegal nationally, via Federal law, for any municipality to ever create or encourage an ISP monopoly; and specifically the exact opposite should occur, competition should be heavily encouraged everywhere. The FCC should be mandated with keeping competition lanes wide open in the ISP market, at all times, across the nation. Standards for how cities and towns handle multiple ISPs for infrastructure should be established. Simply: you shall create no law restricting ISP competition or granting monopoly status, period.

wmf 13y ago
Competition does encourage efficiency, but competing ISPs are starting with lower efficiency because their infrastructure passes so many homes that aren't connected. Pretty much no one understands how Google Fiber can be profitable at its prices, so unless they open their books or someone replicates that business model I'm not going to call it a solution.
kalleboo 13y ago
The OP may be using it wrong, but I still think at this point, the last mile network is a natural monopoly. Where I live, city-owned power companies have built single fiber-to-the-building networks that are the open for ISP competition inside the network (I.e. the ISP you pick just delivers the bandwidth/peering). In my view this is the optimal compromise. This wasn't true just a few years ago when DSL and Cable tech was still competitive, but at this point it seems pretty clear that fiber infrastructure isn't going to be beat.

jlgaddis 13y ago

One other note: the author's school (the entity he brought up the VPN session to) manages many of the largest R&D networks in the country. As such, it is extremely well-connected to, well, just about everything (including Netflix's delivery networks).