6 ms·

I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level

by amalag 12y ago

I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking. Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.

shimshim 12y ago
Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.
- amalag 12y ago
  I bought a VPN and it fixes it. Sad that it is necessary.
  - silverbax88 12y ago
    Which VPN did you go with?
    - amalag 12y ago
      I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
      - djchen 12y ago
        You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.
  - ChuckMcM 12y ago
    I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.
    - Zikes 12y ago
      Can you describe that ssh tunnel issue? Are they monitoring your traffic and actively disrupting anything that looks like ssh?
      - ChuckMcM 12y ago
        Sure, set up an SSH tunnel, make it a SOCKS5 proxy (easy enough to do) to a host in the 'cloud', now proxy you traffic through it (like Netflix). Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all. No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN. It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers. EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.
    - ohashi 12y ago
      When I was on Comcast, I felt like they were doing this to gaming traffic as well. I have no evidence to back it up except anecdotal experience though.
      - Zikes 12y ago
        When I was on AT&T U-Verse they did this with torrents. I replaced my router, OS, and modem thinking the problem lay somewhere on my end, and once I ruled all of that out I called up AT&T to complain and call out the fact that it happened only when I torrented. Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association. I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.
  - IgorPartola 12y ago
    I currently have a 6in4 IPv6 tunnel through Hurricane Electric. At least netflix.com seems to load over IPv6. I wonder if my Netflix experience is fine precisely because my traffic appears to go to HE first, then to Netflix. Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.
    - boredinballard 12y ago
      Interesting. I'm going to have to start trying Netflix and Youtube over IPv6.
- spuiszis 12y ago
  I have Comcast as well and it's gotten because of the deal Netflix struck with them... http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious... EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU https://www.youtube.com/watch?v=fpbOEoRrHyU
mikeash 12y ago
Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.
- lbotos 12y ago
  I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.
  - mikeash 12y ago
    Going beyond that, they typically have a monopoly or duopoly as an ISP, while they're subject to competition as a TV provider. Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services". If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.
- Crito 12y ago
  Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.
- hga 12y ago
  It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"
- fixermark 12y ago
  I'm not a lawyer, but when you put it that way, it almost screams "RICO violation," doesn't it? ;)
induscreep 12y ago
Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.
- bitJericho 12y ago
  Netflix has offered to directly connect to Verizon without the need for Level3 to be in the middle.
  - spyder 12y ago
    and Verizon wrote in their post that they "working" on this: "we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network." But "working" could mean: "waiting for Netflix to pay for it".
    - wmf 12y ago
      Netflix already paid Verizon's ransom and now Verizon is building connects to Netflix but such connections take time to install. Level 3 is arguing over old news at this point, but they're trying to set a precedent to prevent other ISPs from trying the same thing.
  - btgeekboy 12y ago
    This is a good point, but for a different reason: IIRC, Netflix is paying Level3 for connectivity. If Netflix pays Verizon for direct connectivity, Netflix still pays someone for bandwidth, and L3 loses their cut.
- ktsmith 12y ago
  Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.
higherpurpose 12y ago
They just want the same deal as Comcast, that Netflix, in a moment of stupidity, made: http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...
- xxchan 12y ago
  It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.
  - higherpurpose 12y ago
    Well not really. They can just claim that without being paid extra to do so, it wouldn't be cost-effective for them or whatever. I mean that's their public argument anyway - that these services cost them too much and they need to pay up.
msandford 12y ago
Yes, it absolutely is a simple fix. This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one. Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon. And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal. But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know. But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion. But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already. To me it seems like they're abusing monopoly(ish) power.
- Zikes 12y ago
  > Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion. They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.
  - 12y ago
yxhuvud 12y ago
This is not and has never been a technical problem. It is purely in the business and politics areas. One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.

by amalag 12y ago

shimshim 12y ago

Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.

amalag 12y ago
I bought a VPN and it fixes it. Sad that it is necessary.
- silverbax88 12y ago
  Which VPN did you go with?
  - amalag 12y ago
    I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
    - djchen 12y ago
      You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.
- ChuckMcM 12y ago
  I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.
  - Zikes 12y ago
    Can you describe that ssh tunnel issue? Are they monitoring your traffic and actively disrupting anything that looks like ssh?
    - ChuckMcM 12y ago
      Sure, set up an SSH tunnel, make it a SOCKS5 proxy (easy enough to do) to a host in the 'cloud', now proxy you traffic through it (like Netflix). Watch how suddenly you get 'connection reset by peer', look around and discover nobody in your packet path has any cause to reset your traffic. Now do a straight http proxy (varnish works well for this) to the same machine, run it all day night with bits of http traffic. No issues at all. No go back to creating an SSH tunnel between your machine and the endpoint. Run traffic, note the mysterious 'connection reset by peer' when some thing upstream sends your TCP connection a FIN. It could be that their network just has a really hard time with encrypted packets but some how I don't think so. I have tried a variety of port numbers. EDIT: and I went back and verified I still have 'ServerAliveInterval 60' in my config file for all hosts.
  - ohashi 12y ago
    When I was on Comcast, I felt like they were doing this to gaming traffic as well. I have no evidence to back it up except anecdotal experience though.
    - Zikes 12y ago
      When I was on AT&T U-Verse they did this with torrents. I replaced my router, OS, and modem thinking the problem lay somewhere on my end, and once I ruled all of that out I called up AT&T to complain and call out the fact that it happened only when I torrented. Their response was that yes, they do disrupt all torrenting traffic, "because it is illegal and they must comply with copyright holders". Nevermind it was various linux distros I was trying to download and evaluate, all torrenting is automatically evil by association. I had to switch to Cox because they're the only other option in my area. The 250GB data cap on my plan is BS, but at least I can spend that 250GB however I please.
- IgorPartola 12y ago
  I currently have a 6in4 IPv6 tunnel through Hurricane Electric. At least netflix.com seems to load over IPv6. I wonder if my Netflix experience is fine precisely because my traffic appears to go to HE first, then to Netflix. Edit: in the past I have had similarly good experiences with YouTube problems. They seem to go away as soon as you start accessing YouTube over IPv6.
  - boredinballard 12y ago
    Interesting. I'm going to have to start trying Netflix and Youtube over IPv6.
spuiszis 12y ago
I have Comcast as well and it's gotten because of the deal Netflix struck with them... http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious... EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU https://www.youtube.com/watch?v=fpbOEoRrHyU

mikeash 12y ago

Verizon's advantage should be obvious: they also sell TV service which is in direct competition to Netflix. If Netflix works poorly, that's more customers potentially signing up for FiOS TV.

lbotos 12y ago
I feel like this is what people often forget in the net neutrality debate. The pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand) and we need to make sure they aren't rigging the race so to speak.
- mikeash 12y ago
  Going beyond that, they typically have a monopoly or duopoly as an ISP, while they're subject to competition as a TV provider. Much of net neutrality can be reduced to "don't let monopoly ISPs leverage that into a monopoly in other services". If Verizon didn't offer TV, this wouldn't really be a problem. If there were 17 other broadband ISPs to choose from, this wouldn't really be a problem. But Verizon does offer TV and they're one of two choices for me and most of their other customers, so it's a problem.
Crito 12y ago
Furthermore, if Verizon degrades connections to whichever ISPs Netflix uses, then they have leverage to get Netflix to pay Verizon to peer. If they make it so that the only way that Netflix can offer their service is to pay them for it directly, then that is more money for Verizon. Very clear motivation.
hga 12y ago
It's even stated parenthetically in the blog posting when comparing European and US Level 3 and Verizon peering utilization, the former ~18%, and the latter ~100% "(where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses)"
fixermark 12y ago
I'm not a lawyer, but when you put it that way, it almost screams "RICO violation," doesn't it? ;)

induscreep 12y ago

Verizon probably needs to pay Level 3 for the bandwidth - so apart from the cost of the hardware (cards + cables) there is a possibly large cost which is not mentioned in the Level 3 article.

bitJericho 12y ago
Netflix has offered to directly connect to Verizon without the need for Level3 to be in the middle.
- spyder 12y ago
  and Verizon wrote in their post that they "working" on this: "we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network." But "working" could mean: "waiting for Netflix to pay for it".
  - wmf 12y ago
    Netflix already paid Verizon's ransom and now Verizon is building connects to Netflix but such connections take time to install. Level 3 is arguing over old news at this point, but they're trying to set a precedent to prevent other ISPs from trying the same thing.
- btgeekboy 12y ago
  This is a good point, but for a different reason: IIRC, Netflix is paying Level3 for connectivity. If Netflix pays Verizon for direct connectivity, Netflix still pays someone for bandwidth, and L3 loses their cut.
ktsmith 12y ago
Verizon probably doesn't need to pay Level 3 for the bandwidth. Typically peering points like this are settlement free. In other words it's a voluntary exchange of traffic and each side of the line keeps 100% of the revenue from its customers using the interchange.

higherpurpose 12y ago

They just want the same deal as Comcast, that Netflix, in a moment of stupidity, made: http://knowmore.washingtonpost.com/2014/04/25/this-hilarious-graph-of-netflix-speeds-shows-the-importance-of-net-neutrality/ http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...

xxchan 12y ago
It's not stupid at all. By giving into the racket, Netflix demonstrated that it's not a technical issue at all, so that argument no longer flies. It's clear that the ISPs are capable of supporting the service, they just don't want to.
- higherpurpose 12y ago
  Well not really. They can just claim that without being paid extra to do so, it wouldn't be cost-effective for them or whatever. I mean that's their public argument anyway - that these services cost them too much and they need to pay up.

msandford 12y ago

Yes, it absolutely is a simple fix. This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one. Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon. And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal. But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know. But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion. But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already. To me it seems like they're abusing monopoly(ish) power.

Zikes 12y ago
> Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion. They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.
- 12y ago

yxhuvud 12y ago

This is not and has never been a technical problem. It is purely in the business and politics areas. One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.