14 ms·

Verizon's accidental mea culpa

776 points by shimshim 12y ago

shock 12y ago
Seems to have been taken down. Here's the link to the cached version (a bit hard to read): https://webcache.googleusercontent.com/search?q=cache:http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/ https://webcache.googleusercontent.com/search?q=cache:http:/...
- shimshim 12y ago
  strange, wonder why they took it down. edit: this one seems a bit better: http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:DBHDyx7...
- cypher543 12y ago
  The original link still works for me.
  - shimshim 12y ago
    Working for me now as well.
- juliangoldsmith 12y ago
  It's still up for me.
- r0m4n0 12y ago
  Maybe verizon is throttling your connection
- basseq 12y ago
  Works for me. Those who can't see it: are you on Verizon?
  - shock 12y ago
    It's working for me now as well. When I posted the cached version the page on Level3 blog was giving me a 404 although the post was on the "Recent posts" list. I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.
amalag 12y ago
I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking. Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.
- shimshim 12y ago
  Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.
  - amalag 12y ago
    I bought a VPN and it fixes it. Sad that it is necessary.
    - silverbax88 12y ago
      Which VPN did you go with?
      - amalag 12y ago
        I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
        djchen 12y ago
        You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.
    - ChuckMcM 12y ago
      I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.
bitJericho 12y ago
"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it." That's what I would call a mortal wound to Verizon's arguments.
- ohashi 12y ago
  I love this. All of what I presume was happening behind closed doors is finally getting put into the light. If only there were serious options for consumers to act on based on the information that is coming out publicly.
  - druidsbane 12y ago
    L3 as a consumer broadband provider. Cut out all the other middle-men.
    - joepie91_ 12y ago
      That would be a terrible idea (currently). I don't think they have any last-mile infrastructure.
- sheetjs 12y ago
  Clever technique to draw attention away from what is most likely a contractual dispute. Verizon's argument is clear: > Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
  - chimeracoder 12y ago
    > (in this case, since Level3 is pushing more data, they would be the ones paying). That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers. All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern. Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).
phkahler 12y ago
I've always been suspect of cable and wireless internet connections. Anything that shares bandwidth with all subscribers is going to get maxed out if a large number of customers start streaming video. Is it possible that Verison (and Comcast and etc...) know that opening up the gate to the outside world will crush their network under the increased load? I like the milk-our-monopoly-on-the-last-mile theory as much as the next guy, but is it even possible for the technology used to deliver the necessary bandwidth if they open up the pipe?
- shimshim 12y ago
  Even so, as a paying customer I would expect them to calculate for this and know how to scale.
  - phkahler 12y ago
    My point is was that spectrum doesn't scale. There's X gigabits per second available over the air, and a similar X over a cable. For cable companies they can push equipment closer to the home and have fewer subscribers on the same line. I suppose Verizon could do the same with more towers, but imagine doubling the number of towers to support double the bandwidth (or something like that). As people have pointed out though, the congestion is specific to connections that carry netflix.
    - PeterisP 12y ago
      There's a limit for mobile bandwidth, where also the number of towers won't help; but cable is different - sure, at one point they have to put proper fiber to the home (and it's economically feasible to do so, if there's actual competition instead of monopolistic ability to charge huge prices for weak service), but after that you can have as much bandwidth as your routers can service.
- ktsmith 12y ago
  Oversubscribing lines has been one of the ways to keep ISPs profitable for years. Streaming video absolutely cuts into the ability to do this of course but I doubt it's the primary issue here. Verizon has every incentive to not add more bandwidth to their peering points with Level 3 and try to get more money from Netflix directly.
bcRIPster 12y ago
Let's not forget that those providers that Netflix chooses to use are global edge service providers for high bandwidth streaming content services. Cogent, Level 3, Limelight, etc... Their whole business is providing high capacity back-end network transit which they do well, otherwise they wouldn't be in business for very long. High-five Level 3 for calling B.S. on this!
- MichaelGG 12y ago
  Funny you'd mention Cogent and L3 together, seeing as L3 has done worse to Cogent (depeering) for the exact same reason Verizon is using (unbalanced traffic).
  - bcRIPster 12y ago
    Oh I didn't say they weren't bastards. That's a whole other discussion.
mikecb 12y ago
Is it just me or is this not at all inconsistent with Verizon's arguments? They've been saying the whole time that this link is meant to be balanced with roughly equal upstream and downstream, and they showed it was not. What am I missing?
- Crito 12y ago
  Residential ISPs have pretty much never, in the history of ever, not had asymmetrical traffic. Netflix may have aggravated that, but they did not change that.
  - RKearney 12y ago
    What made me laugh the most was when Comcast posted about a peering dispute with Level3[0] >Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast? Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps) Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3? [0] http://corporate.comcast.com/comcast-voices/20-qs-with-accurate-as-about-level-3s-peering-dispute http://corporate.comcast.com/comcast-voices/20-qs-with-accur...
  - mikecb 12y ago
    And I don't know how Verizon's backbone is laid out. But they have transit and end-user traffic. Regardless, if Level3 keeps insisting this is a peering point, and then demonstrating that it's more like a transit to ISP link. Which ISPs pay for.
- msandford 12y ago
  You've missed the part where they're trying to double-bill for the same network traffic. Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix. Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending. But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill. Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.
deleted 12y ago
[deleted]
trjordan 12y ago
The claim from Verizon that still stands after this blog post is the fairness argument to peering. If you peer between symmetric networks, it's typically free. That's not the case here, so Verizon wants to be paid for the asymmetry. I actually don't know. Is that legitimate?
- bhouston 12y ago
  Consumer internet providers will always have assymmetry because people generally are consumers of content, not creators or distributors. Remember that standard home internet connections have assymmetric download/upload speeds and also ban consumers from running hosting via their consumer grade internet connection.
- davej 12y ago
  No, it's not. Their customers are paying Verizon money for access to the internet, their customers are accessing Netflix.
- wil421 12y ago
  Level 3 is a tier 1 provider so in theory they should be peering for "free" with other Tier 1 providers such as Verizon. From Wikipedia: "However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements." http://en.wikipedia.org/wiki/Tier_1_network http://en.wikipedia.org/wiki/Tier_1_network
- masklinn 12y ago
  > Is that legitimate? Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p. Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot. VZ just wants to double-bill.
- rhino369 12y ago
  The issue is that Verizon is both a transit provider and a last mile ISP. For transit providers, asymmetry = Paid. For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit. Since verizon is a T1 transit provider, it doesn't want to give it's service away for free. I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement). I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all. I'm not sure what is actually happening. I'd imagine a mix of both.
wil421 12y ago
It doesnt matter what content is coming down the wire, if I pay for internet I expect to be able to access or connect to anything thats not forbidden in the EULA. With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?
- MichaelGG 12y ago
  Cool! Now tell Comcast you want to download 300GB from my server, which I've got connected on a single 300 baud channel. Will they come give me a free 1GE connection?
  - azernik 12y ago
    Nope. Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP. For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.
  - wil421 12y ago
    It doesnt work like that because Netflix has already paid level 3 who does most of the heavy lifting. Similar to if you had a 10GE connection at your sever but comcast is still running a 300 baud channel on the connection they have to your server.
    - MichaelGG 12y ago
      OK, so I'll buy a 10G port to another random ISP. And they'll go to the ICX and say "hey Verizon, hit me up with the 10G port". Oh wait, it doesn't work like that. It works with peering arrangements, hmm.
      - wil421 12y ago
eyeareque 12y ago
He makes great points. But this is what the verizon marketing team is paid to repond to. They'll come up with something, just wait for it.
staunch 12y ago
This is the entire net neutrality debate in one simple blog post. Verizon is selling internet access to their users but not actually providing real access!
troels 12y ago
One question: There aren't that many large edge providers around. What would Verizon do if Netflix signed a contract with a dozen of the larger ones and routed their streams through whichever had the best connectivity for a given endpoint? If Verizon throttle all of them, they are going to get a lot of collateral - essentially degrading the Internet at large for their customers. Or would they just do packet level filtering then?
- josho 12y ago
  I've wondered this as well. I suspect that it would be cost prohibitive for Netflix to have a contract with several edge providers like Level 3. I'm certain that Verizon's price demands to Netflix would be cheaper than this option. Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).
  - wmf 12y ago
    I think the costs work out differently at terabit scale. Sending a terabit split across two backbones may be the same cost or cheaper than sending it over a single backbone.
  - troels 12y ago
    True. But I would assume that the contract is largely a pay per usage or at least expected usage. So if they just make an estimate of how many Verizon customers they have and buy that block from someone else than Level 3. Then route all that traffic through the other provider, while keep using Level 3 for the majority of their traffic. Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there. In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?
gcb0 12y ago
i recently gave up my netflix and amazon prime because i was tired of streaming hanging up in the middle of an episode/movie. it was just annoying and embarrassing if i had guests. now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem. but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon? as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?
- FireBeyond 12y ago
  "but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?" Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.
ninv 12y ago
Greed! It would not stop here. Today it is Netflix and youtube, tomorrow it will be image heavy sites. Then they will go after google, mail provider and bank sites etc.
ajmurmann 12y ago
The Internet is a crucial part of our infrastructure and a important backbone of our economy. If there is any indication that someone like Verizon in this case is tempering with it intentionally, they should be charged with terrorism. This is no different than someone going out and sabotaging Hoover damn on purpose. In fact it's worth!
BryantD 12y ago
Point of information about Verizon as a purely residential carrier: Verizon owns MCI/Worldcom and Terremark. They're not just a residential carrier. I don't have any idea what the ratio of residential to business traffic is, of course, but I think Verizon is a real Tier 1 carrier. Happy to have someone with more definite data chime in if I'm wrong.
- msandford 12y ago
  They are but the residential and commercial sides of the business are VERY different: https://news.ycombinator.com/item?id=8050089 https://news.ycombinator.com/item?id=8050089
  - BryantD 12y ago
    Do they use different backbones? Which side of the business runs the interconnects?
    - msandford 12y ago
      Verizon might technically be one company but in reality there are several business units which are actually separate companies in all but name. When you buy business backhaul from Verizon you get guaranteed bandwidth provisioned on their network from point A to B. They don't over provision because they can't: contractually. If they don't meet their SLA the customer is out of the contract and can go elsewhere. On the residential side things are very different. We have no SLA other than "pfft whatever we'll roll a truck next Thursday I guess" Verizon might share long distance backhaul between residential and commercial sides of the company but if so it's done with strict quotas to ensure that the commercial side of things isn't impacted. It's not as though the core network division lets everyone have a free-for-all on the available bandwidth through the core. If I had to guess I would venture that the consumer group either runs largely or entirely separate equipment. They might share fibers but I'm not all that confident about that. They're very, very close to entirely separate businesses.
adamman 12y ago
Doesn't Verizon realize they lose customers for crap like this? My phone contract with Verizon is up in November. I can't wait to be done with them.
revelation 12y ago
Theres a common theme in this discussion. We're stuck discussing analogies. Instead, we should be reasoning from "first principles" as Musk suggests. If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous. There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.
- bitJericho 12y ago
  I'm pretty sure that was an analogy! (A good one though)
- deleted 12y ago
  [deleted]
knodi 12y ago
If only we have a government body that did something about stuff like this... o wait FCC... o wait :(
abalone 12y ago
Summary: The bottleneck is the the interconnection point. Netflix is saturating the interconnection point, and Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic". But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.
- msandford 12y ago
  Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack. This is Verizon wanting to double-bill for traffic. Plain and simple. > Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic". That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!
  - res0nat0r 12y ago
    > Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack. All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering. It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.
    - gergles 12y ago
      You don't peer with residential ISPs, because they aren't backbone ISPs. Residential ISPs need to buy transit from Level 3. If anything, Verizon should be L3s customer, and L3 giving them SFP is doing Verizon a favor.
drderidder 12y ago
How did people think this wouldn't happen when ISP's are allowed to be content providers, too. Of course Verizon wants to choke Netflix and promote their own service instead. Letting cable and content providers be ISPs was always a bad idea.
macspoofing 12y ago
Great post. Frankly Level3 should be pissed off. Verizon essentially called them out for providing crap service when in fact they are the ones responsible.
slyrus 12y ago
More like an accidental "Fuck you" than an accidental "mea culpa".
brownbat 12y ago
What would happen if Netflix stopped allowing access to its service to Verizon customers in areas where there's another provider? Netflix could argue that it doesn't want to provide an inferior product if there's a choice, and maybe leave a number on the homepage for the cable company in your area. I wonder if Netflix or Verizon would lose more customers...
- wpietri 12y ago
  Then a) Netflix becomes the bad guy, b) they lose money from those customers leaving, and c) they no longer have the opportunity to influence those people. They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.
- deleted 12y ago
  [deleted]
ck2 12y ago
This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. This needs to be put in front of the FCC since it is the bare minimum language they should be able to understand.
exabrial 12y ago
I think a natural solution to this is to have Netflix charge all Verizon customers a $5/mo surcharge to pay for 4 more 10gbps ports. But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?

776 points by shimshim 12y ago

shock 12y ago

Seems to have been taken down. Here's the link to the cached version (a bit hard to read): https://webcache.googleusercontent.com/search?q=cache:http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/ https://webcache.googleusercontent.com/search?q=cache:http:/...

shimshim 12y ago
strange, wonder why they took it down. edit: this one seems a bit better: http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:DBHDyx7...
cypher543 12y ago
The original link still works for me.
- shimshim 12y ago
  Working for me now as well.
juliangoldsmith 12y ago
It's still up for me.
r0m4n0 12y ago
Maybe verizon is throttling your connection
basseq 12y ago
Works for me. Those who can't see it: are you on Verizon?
- shock 12y ago
  It's working for me now as well. When I posted the cached version the page on Level3 blog was giving me a 404 although the post was on the "Recent posts" list. I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.

amalag 12y ago

I have to agree with this assessment. If Verizon is claiming they have green in their network with 34% utilization to Level 3, then the connection between Level 3 and Verizon is lacking. Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.

shimshim 12y ago
Its not always great on Comcast either. It has gotten better over the last 6 months though, I will admit.
- amalag 12y ago
  I bought a VPN and it fixes it. Sad that it is necessary.
  - silverbax88 12y ago
    Which VPN did you go with?
    - amalag 12y ago
      I used privateinternetaccess, i am happy with it. I don't even use it all the time but the speed tests hold up against my 50Mb/s connection. Hulu has blocked it if that is a concern, there is some other VPN that has more egress IP's that works with Hulu, but I didn't know about it and it was more expensive anyway.
      - djchen 12y ago
        You can run your own VPN on a VPS and have a dedicated IP without needing to worry about Hulu getting blocked.
  - ChuckMcM 12y ago
    I did this on Comcast as well, but its interesting to note they do shoot downs of ssh tunnels. Using IPSEC hasn't been an issue though.

bitJericho 12y ago

"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it." That's what I would call a mortal wound to Verizon's arguments.

ohashi 12y ago
I love this. All of what I presume was happening behind closed doors is finally getting put into the light. If only there were serious options for consumers to act on based on the information that is coming out publicly.
- druidsbane 12y ago
  L3 as a consumer broadband provider. Cut out all the other middle-men.
  - joepie91_ 12y ago
    That would be a terrible idea (currently). I don't think they have any last-mile infrastructure.
sheetjs 12y ago
Clever technique to draw attention away from what is most likely a contractual dispute. Verizon's argument is clear: > Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
- chimeracoder 12y ago
  > (in this case, since Level3 is pushing more data, they would be the ones paying). That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers. All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern. Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).

phkahler 12y ago

I've always been suspect of cable and wireless internet connections. Anything that shares bandwidth with all subscribers is going to get maxed out if a large number of customers start streaming video. Is it possible that Verison (and Comcast and etc...) know that opening up the gate to the outside world will crush their network under the increased load? I like the milk-our-monopoly-on-the-last-mile theory as much as the next guy, but is it even possible for the technology used to deliver the necessary bandwidth if they open up the pipe?

shimshim 12y ago
Even so, as a paying customer I would expect them to calculate for this and know how to scale.
- phkahler 12y ago
  My point is was that spectrum doesn't scale. There's X gigabits per second available over the air, and a similar X over a cable. For cable companies they can push equipment closer to the home and have fewer subscribers on the same line. I suppose Verizon could do the same with more towers, but imagine doubling the number of towers to support double the bandwidth (or something like that). As people have pointed out though, the congestion is specific to connections that carry netflix.
  - PeterisP 12y ago
    There's a limit for mobile bandwidth, where also the number of towers won't help; but cable is different - sure, at one point they have to put proper fiber to the home (and it's economically feasible to do so, if there's actual competition instead of monopolistic ability to charge huge prices for weak service), but after that you can have as much bandwidth as your routers can service.
ktsmith 12y ago
Oversubscribing lines has been one of the ways to keep ISPs profitable for years. Streaming video absolutely cuts into the ability to do this of course but I doubt it's the primary issue here. Verizon has every incentive to not add more bandwidth to their peering points with Level 3 and try to get more money from Netflix directly.

bcRIPster 12y ago

Let's not forget that those providers that Netflix chooses to use are global edge service providers for high bandwidth streaming content services. Cogent, Level 3, Limelight, etc... Their whole business is providing high capacity back-end network transit which they do well, otherwise they wouldn't be in business for very long. High-five Level 3 for calling B.S. on this!

MichaelGG 12y ago
Funny you'd mention Cogent and L3 together, seeing as L3 has done worse to Cogent (depeering) for the exact same reason Verizon is using (unbalanced traffic).
- bcRIPster 12y ago
  Oh I didn't say they weren't bastards. That's a whole other discussion.

mikecb 12y ago

Is it just me or is this not at all inconsistent with Verizon's arguments? They've been saying the whole time that this link is meant to be balanced with roughly equal upstream and downstream, and they showed it was not. What am I missing?

Crito 12y ago
Residential ISPs have pretty much never, in the history of ever, not had asymmetrical traffic. Netflix may have aggravated that, but they did not change that.
- RKearney 12y ago
  What made me laugh the most was when Comcast posted about a peering dispute with Level3[0] >Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast? Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps) Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3? [0] http://corporate.comcast.com/comcast-voices/20-qs-with-accurate-as-about-level-3s-peering-dispute http://corporate.comcast.com/comcast-voices/20-qs-with-accur...
- mikecb 12y ago
  And I don't know how Verizon's backbone is laid out. But they have transit and end-user traffic. Regardless, if Level3 keeps insisting this is a peering point, and then demonstrating that it's more like a transit to ISP link. Which ISPs pay for.
msandford 12y ago
You've missed the part where they're trying to double-bill for the same network traffic. Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix. Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending. But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill. Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.

deleted 12y ago

[deleted]

trjordan 12y ago

The claim from Verizon that still stands after this blog post is the fairness argument to peering. If you peer between symmetric networks, it's typically free. That's not the case here, so Verizon wants to be paid for the asymmetry. I actually don't know. Is that legitimate?

bhouston 12y ago
Consumer internet providers will always have assymmetry because people generally are consumers of content, not creators or distributors. Remember that standard home internet connections have assymmetric download/upload speeds and also ban consumers from running hosting via their consumer grade internet connection.
davej 12y ago
No, it's not. Their customers are paying Verizon money for access to the internet, their customers are accessing Netflix.
wil421 12y ago
Level 3 is a tier 1 provider so in theory they should be peering for "free" with other Tier 1 providers such as Verizon. From Wikipedia: "However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements." http://en.wikipedia.org/wiki/Tier_1_network http://en.wikipedia.org/wiki/Tier_1_network
masklinn 12y ago
> Is that legitimate? Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p. Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot. VZ just wants to double-bill.
rhino369 12y ago
The issue is that Verizon is both a transit provider and a last mile ISP. For transit providers, asymmetry = Paid. For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit. Since verizon is a T1 transit provider, it doesn't want to give it's service away for free. I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement). I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all. I'm not sure what is actually happening. I'd imagine a mix of both.

wil421 12y ago

It doesnt matter what content is coming down the wire, if I pay for internet I expect to be able to access or connect to anything thats not forbidden in the EULA. With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?

MichaelGG 12y ago
Cool! Now tell Comcast you want to download 300GB from my server, which I've got connected on a single 300 baud channel. Will they come give me a free 1GE connection?
- azernik 12y ago
  Nope. Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP. For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.
- wil421 12y ago
  It doesnt work like that because Netflix has already paid level 3 who does most of the heavy lifting. Similar to if you had a 10GE connection at your sever but comcast is still running a 300 baud channel on the connection they have to your server.
  - MichaelGG 12y ago
    OK, so I'll buy a 10G port to another random ISP. And they'll go to the ICX and say "hey Verizon, hit me up with the 10G port". Oh wait, it doesn't work like that. It works with peering arrangements, hmm.
    - wil421 12y ago

eyeareque 12y ago

He makes great points. But this is what the verizon marketing team is paid to repond to. They'll come up with something, just wait for it.

staunch 12y ago

This is the entire net neutrality debate in one simple blog post. Verizon is selling internet access to their users but not actually providing real access!

troels 12y ago

One question: There aren't that many large edge providers around. What would Verizon do if Netflix signed a contract with a dozen of the larger ones and routed their streams through whichever had the best connectivity for a given endpoint? If Verizon throttle all of them, they are going to get a lot of collateral - essentially degrading the Internet at large for their customers. Or would they just do packet level filtering then?

josho 12y ago
I've wondered this as well. I suspect that it would be cost prohibitive for Netflix to have a contract with several edge providers like Level 3. I'm certain that Verizon's price demands to Netflix would be cheaper than this option. Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).
- wmf 12y ago
  I think the costs work out differently at terabit scale. Sending a terabit split across two backbones may be the same cost or cheaper than sending it over a single backbone.
- troels 12y ago
  True. But I would assume that the contract is largely a pay per usage or at least expected usage. So if they just make an estimate of how many Verizon customers they have and buy that block from someone else than Level 3. Then route all that traffic through the other provider, while keep using Level 3 for the majority of their traffic. Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there. In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?

gcb0 12y ago

i recently gave up my netflix and amazon prime because i was tired of streaming hanging up in the middle of an episode/movie. it was just annoying and embarrassing if i had guests. now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem. but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon? as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?

FireBeyond 12y ago
"but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?" Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.

ninv 12y ago

Greed! It would not stop here. Today it is Netflix and youtube, tomorrow it will be image heavy sites. Then they will go after google, mail provider and bank sites etc.

ajmurmann 12y ago

The Internet is a crucial part of our infrastructure and a important backbone of our economy. If there is any indication that someone like Verizon in this case is tempering with it intentionally, they should be charged with terrorism. This is no different than someone going out and sabotaging Hoover damn on purpose. In fact it's worth!

BryantD 12y ago

Point of information about Verizon as a purely residential carrier: Verizon owns MCI/Worldcom and Terremark. They're not just a residential carrier. I don't have any idea what the ratio of residential to business traffic is, of course, but I think Verizon is a real Tier 1 carrier. Happy to have someone with more definite data chime in if I'm wrong.

msandford 12y ago
They are but the residential and commercial sides of the business are VERY different: https://news.ycombinator.com/item?id=8050089 https://news.ycombinator.com/item?id=8050089
- BryantD 12y ago
  Do they use different backbones? Which side of the business runs the interconnects?
  - msandford 12y ago
    Verizon might technically be one company but in reality there are several business units which are actually separate companies in all but name. When you buy business backhaul from Verizon you get guaranteed bandwidth provisioned on their network from point A to B. They don't over provision because they can't: contractually. If they don't meet their SLA the customer is out of the contract and can go elsewhere. On the residential side things are very different. We have no SLA other than "pfft whatever we'll roll a truck next Thursday I guess" Verizon might share long distance backhaul between residential and commercial sides of the company but if so it's done with strict quotas to ensure that the commercial side of things isn't impacted. It's not as though the core network division lets everyone have a free-for-all on the available bandwidth through the core. If I had to guess I would venture that the consumer group either runs largely or entirely separate equipment. They might share fibers but I'm not all that confident about that. They're very, very close to entirely separate businesses.

adamman 12y ago

Doesn't Verizon realize they lose customers for crap like this? My phone contract with Verizon is up in November. I can't wait to be done with them.

revelation 12y ago

Theres a common theme in this discussion. We're stuck discussing analogies. Instead, we should be reasoning from "first principles" as Musk suggests. If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous. There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.

bitJericho 12y ago
I'm pretty sure that was an analogy! (A good one though)
deleted 12y ago
[deleted]

knodi 12y ago

If only we have a government body that did something about stuff like this... o wait FCC... o wait :(

abalone 12y ago

Summary: The bottleneck is the the interconnection point. Netflix is saturating the interconnection point, and Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic". But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.

msandford 12y ago
Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack. This is Verizon wanting to double-bill for traffic. Plain and simple. > Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic". That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!
- res0nat0r 12y ago
  > Nope, Verizon's customers are causing the interconnection point to be saturated. Netflix doesn't send data unless Netflix's customers request it. If they sent data without requests Verizon could (and probably would) interpret that as some kind of a network attack. All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering. It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.
  - gergles 12y ago
    You don't peer with residential ISPs, because they aren't backbone ISPs. Residential ISPs need to buy transit from Level 3. If anything, Verizon should be L3s customer, and L3 giving them SFP is doing Verizon a favor.

drderidder 12y ago

How did people think this wouldn't happen when ISP's are allowed to be content providers, too. Of course Verizon wants to choke Netflix and promote their own service instead. Letting cable and content providers be ISPs was always a bad idea.

macspoofing 12y ago

Great post. Frankly Level3 should be pissed off. Verizon essentially called them out for providing crap service when in fact they are the ones responsible.

slyrus 12y ago

More like an accidental "Fuck you" than an accidental "mea culpa".

brownbat 12y ago

What would happen if Netflix stopped allowing access to its service to Verizon customers in areas where there's another provider? Netflix could argue that it doesn't want to provide an inferior product if there's a choice, and maybe leave a number on the homepage for the cable company in your area. I wonder if Netflix or Verizon would lose more customers...

wpietri 12y ago
Then a) Netflix becomes the bad guy, b) they lose money from those customers leaving, and c) they no longer have the opportunity to influence those people. They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.
deleted 12y ago
[deleted]

ck2 12y ago

This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. This needs to be put in front of the FCC since it is the bare minimum language they should be able to understand.

exabrial 12y ago

I think a natural solution to this is to have Netflix charge all Verizon customers a $5/mo surcharge to pay for 4 more 10gbps ports. But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?