12 ms·

Why is Netflix Buffering? Dispelling the Congestion Myth

106 points by danielsamuels 12y ago

jrs235 12y ago
Verizon's customers request an unprecedented amount of data and traffic from Netflix. This is Verizon's problem, not Netflix's. Verizon should be working with Netflix on an amicable solution to perhaps cache data inside Verizon's network... O What do you know! Netflix already has a program for that: https://www.netflix.com/openconnect https://www.netflix.com/openconnect
- diminoten 12y ago
  To be fair, this problem is non-existant today, because Netflix has entered into payed peering agreements with Verison/Comcast/other ISPs. They more or less acquiesced. They're just not happy about it.
rnovak 12y ago
They can claim whatever they want, but when you tunnel Netflix traffic through a VPN so all the ISP see's is encrypted traffic and bandwidth issues disappear, it removes all mystery. I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish. considering what we're paying for broadband, we should be getting what we pay for without resorting to obfuscating our data.
- kevinskii 12y ago
  I tried the same thing a couple of years ago and observed the same, and then switched providers.
  - cordite 12y ago
    If only more had that option.
    - pcl 12y ago
      In SF, WebPass (and MonkeyBrains, from what I hear) are great alternative ISPs. Both do point-to-point wireless. My building shares a symmetric 100Mbps connection to WebPass, and I pretty reliably see 30-40Mbps on my wired network. And I pay significantly less than I had previously paid Comcast. (However, I never used the TV component of my Comcast offer, so the cost structure is presumably different for people who do want cable TV.) I understand that point-to-point wireless is more realistic in an urban environment than in the suburbs or a more rural setting. Do other cities have good alternate ISPs, or are SF residents somewhat unique in their appetite to try alternate providers?
      - sparkman55 12y ago
        Point-to-Point (Fixed) Wireless is actually also an attractive solution in rural / semi-rural settings as well. If the distances are moderate (with line-of-sight) but demand is too sparse to warrant the cost of running coaxial / fiber (and distances are too far for DSL), a common alternative on the high end is fixed microwave, and on the low end Verizon LTE. A good example is the Ames, Iowa area, where an ISP has set up a tower with good line of sight, and a bunch of directional antennas both on the tower and at customers locations. This setup provides great service out to ~15 miles or so, enough to cover the outskirts of this small town. Some issues with this setup: storms cause 'rain fade' which blocks the signal, and the wireless spectrum does not scale to too many users. Also, there's a finicky antenna on the roof that can become misaligned in storms.
      - bicknergseng 12y ago
        My building in SF is locked in to ATT Uverse. I'd kill for WP or MB.
    - Someone1234 12y ago
      That's really the key problem here. There is lack of choice in the US and Canada. Everyone loves to drum their chest about capitalism but in this case it isn't working, and ISPs have clearly colluded to divvy up the territories so they aren't directly competing. The internet has become a public utility just like electricity, water, or the roads. Businesses and private citizens depend on it day to day, and economic growth is hampered by poor performance in this area (e.g. new types of businesses that depend on bandwidth, like a Netflix competitor, cannot exist if the pipes are slow). In my view the raw cable and fibre should be confiscated from the big three by the government using eminent domain, and then rented out to competing ISPs (including the big three) on even footing. The rent would pay to maintain the cables/fibre and deploy more network, and the ISPs could add a premium before reselling it to end users to cover support costs and maintaining equipment (e.g. routers, edge connections, etc). Basically make the final mile public property. It would make real competition as smaller ISPs can now compete with large ones, and there isn't regularly problems for these companies (e.g. getting permits to place new cables).
  - seanflyon 12y ago
    Assuming your new provider did not have the problem, which ISP did you switch to?
- random28345 12y ago
  > I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish. That's not inconsistent with Verizon's statement. Verizon is saying, "we won't accept more traffic directly from Netflix (via their transit providers) until we get more money!". By using a VPN, you're avoiding that direct connection, and traversing a less-loaded link.
  - swasheck 12y ago
    Exactly. They took Netflix's bait with the "congestion" message. They're demonstrating that their network isn't congested and now we're closer to seeing that they really are attempting to charge multiple customers for the same traffic.
- agwa 12y ago
  That's because your VPN providers are no doubt using a different transit provider than Netflix (Level 3) so your traffic is bypassing the congested peering points. The peering disputes are affecting not just Netflix traffic, but traffic between Verizon and any site using Level 3 for transit. (Not to take the blame away from Verizon. I'd love to see them try to use their logic to explain why other, non-Netflix services work like crap from Verizion.)
  - joeframbach 12y ago
    Isn't the Netflix-to-Verizon transit something that we already pay Verizon to maintain?
    - maxsilver 12y ago
      Yes, but Verizon also wants Netflix to pay them for that same transit. That's what the whole blog article is about. Verizon trying to invent valid reasons for why they want Netflix to pay them.
      - jpgvm 12y ago
        They are not inventing reasons. Netflix is serving the content, Netflix pays for it to move. Simple as that. The issue is Layer 3 is not buying enough transit from Verizon to match the disparity in the traffic it wants to send beyond what is covered by settlement free peering. If you look at the diagram on the Verizon post you can see the link is saturated in one direction and only at 30% utilization in the other direction. This is the DEFINITION of transit. The fact Netflix is even blaming Verizon for this is ridiculous.
        Terretta 12y ago
        > If you look at the diagram on the Verizon post The diagram is wrong. The thing they colored red is not red. The wire is fine. The box inside Verizon's network has a network card in it that's oversubscribed. A slice of their box should be red. Btw, the definition I'm familiar with for "transit" is crossing a network backbone, it's neither serving, nor consuming. The traffic "transits" your network, on its way from somewhere to somewhere.
- azurelogic 12y ago
  I had this same experience with Comcast. See also Level 3's blog post calling Verizon and other ISPs out on this BS: http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte...
  - tempestn 12y ago
    Exactly. Verizon can try to spin this as "Netflix is pushing this traffic on us through inferior intermediate links," but what's really happening is that Verizon's customers are requesting the traffic. The standard response when one of their border links is congested is to work with the peer to upgrade it. Level3 is willing to do so. Verizon would rather try to squeeze money out of Netflix for an unnecessary "direct link". Hopefully people see through this PR.
  - nathanyz 12y ago
    Yep this is exactly the fast lane that we don't want. The lane is not any faster, just less traffic, says Verizon--really, that seems like a fast lane to me.
    - deleted 12y ago
      [deleted]
    - cmiles74 12y ago
      Since Verizon is eliminating so much traffic, those FIOS links are free for downloading all kinds of content their customers don't want.
- digitalpacman 12y ago
  I want Verizon to reply exactly to this more than I want them to reply to anything Netflix has said.
- kyberias 12y ago
  Assuming you connected your Verizon IP to a VPN server that resides in a non-Verizon network, that test is insufficient to deduce that the ISP looks at the data. As Verizon is saying, Netflix does not pay enough for the data that they send. They don't need to inspect the data. By connecting your VPN to a non-Verizon network, Verizon sees the Netflix stream coming from that network (not Netflix).
  - dasil003 12y ago
    How does Verizon know if Netflix is paying enough for the data they send? They are not Netflix's ISP.
- gabriel34 12y ago
  Couldn't it be that your VPN provider has better/less crowded connections to both Verizon and Netflix? I can see how using a lower grade link to a main source of traffic outside a network can actually shape the traffic and reduce the workload on said network. If that is not Verizon's doing and if there is some reason why Verizon is not connected to other Netflix's links then I can see it as being out of Verizon hands. It is more probable, though that the limitations in interconnection between Verizon and Netflix are intentional so as to avoid high utilization of Verizon's network.
sergiotapia 12y ago
It's so strange, I live in Bolivia and Netflix never buffers for me at all. I have a laughable 200kbps real download speed (1768kbps plan) and I only wait for Netflix to load for about 2 seconds on my PS4 and off I go watching on 480SD quality. I would expect the US to be much much better.
- skelsey 12y ago
  No one in the US wants to watch 480 video.
  - maxsilver 12y ago
    Or can. My 5mbps AT&T DSL struggles to load Hulu and Netflix on a daily basis, even at just 480p
  - lostcolony 12y ago
    I do. Comcast. Data cap. Of course, what I would -prefer- is to ditch Comcast, and yes, watch at a higher resolution, and I will at the first available opportunity. But can't. Because monopoly.
    - swasheck 12y ago
      nonsense. there is no monopoly in capitalism. /sarcasm
      - adventured 12y ago
        If there was Capitalism you'd actually be able to compete.
        swasheck 12y ago
        There is Capitalism ... but it's been corrupted. Economic systems are morally neutral. How people employ them determines the morality. The U.S. is, in general, pushing Capitalism to its logical end, in which the system consumes itself in the name of its constituents' "best interests."
jflowers45 12y ago
Let the verbal battle begin!
wavefunction 12y ago
I know who I believe, and it's not Verizon!!!
dcc1 12y ago
LOL taking 30 seconds+ to even load their page
- jrs235 12y ago
  They must be using Netflix's network.
- RandomSpamMan 12y ago
  Doesn't even load for me. http://i.imgur.com/wT7zV3f.png http://i.imgur.com/wT7zV3f.png
- josefresco 12y ago
  Yeah the site/page has been down/slow. Getting ISE 500 errors.
- eddieh 12y ago
  The first time the page failed to load, the second time it took 52.76 seconds according to the Chrome Developer Tools. I'm on a university network that is super fast.
esoterae 12y ago
How apropos that the link is showing unavailable after timeout & errors after only 17 minutes of being posted. <Verizon> It's definitely not us. Yep. Not it.
idiot900 12y ago
A well written PR piece, but no new insights here. Verizon feels it is not Verizon's job as an Internet service provider to provide adequate service to parts of the Internet that its customers are paying them to access. This wouldn't be a problem if broadband access were actually a competitive market - customers would simply switch to a provider that had taken the initiative to give its customers what they want.
- seanflyon 12y ago
  Thank you. While almost all the comments are saying the same thing, I think you stated it best.
angersock 12y ago
Somewhat ironically, the page is taking forever to load.
- cdnsteve 12y ago
  500 Internal Server Error for me right now
JimmaDaRustla 12y ago
Why is the "transit providers" split into two segments - inbound and outbound? Wouldn't it be the same logical segment and be impacted based on data in both directions?
- random28345 12y ago
  > Why is the "transit providers" split into two segments - inbound and outbound? Because they are trying to illustrate how unbalanced the traffic flow is in each direction, to bolster their argument that Verizon shouldn't have to deliver the streams that Verizon customers requested until Netflix gives them more money.
LandoCalrissian 12y ago
Long story short, Level 3 won't pay us for peering agreements so it's their fault.
- techwizrd 12y ago
  The most incredible part is that the facts and anecdotes presented so far don't agree with Verizon's tale, so instead of Verizon decided to respond with a fluffy PR piece instead of fixing the problem or countering with facts. Verizon seems completely unwilling to take any blame in this and does not seem to have their customer's best interest at heart. There are no new facts in this article other than a useless infographic and repeated claims that Netflix is saturating their link with content Verizon customers are asking for.
  - yaur 12y ago
    The only new fact I see here is that link between L3 and Verizon is not symmetric. That's not surprising really, but it means that there is already traffic asymmetry baked into whatever peering agreement they have today.
- fryguy 12y ago
  It seems like Netflix is the one providing value to the Verizon network so it should be the other way around. It would be like someone saying "I'm not downloading these files from a webserver because they're not paying me to download it."
pizza 12y ago
Can anyone recommend networking resources analogous to Learn C the Hard Way or Dive into Python?
- thrush 12y ago
  High Performance Browser Networking by Ilya Grigorik is pretty solid, and free online. http://chimera.labs.oreilly.com/books/1230000000545 http://chimera.labs.oreilly.com/books/1230000000545
- bqe 12y ago
  Beej's Guide to Network Programming is still the best. http://beej.us/guide/bgnet/ http://beej.us/guide/bgnet/
malchow 12y ago
"Netflix sends out an unprecedented amount of traffic" is one of several misleading locutions in this piece. Verizon customers request these packets. The ISPs' problem is that they aren't willing to change their pitch to customers. As long as their product is "internet service at ~x gbps," they are on the wrong side of this issue. So they go ransoming the content folks. The funny thing is that the dream world of the big modern ISPs would appear to be something like AOL circa 1994, where you can have "the internet," but it's a cheap lo-fi interconnect [1], and what they mainly want you to consume is a bunch of low-quality in-house content channels. [2] [1] http://images.appleinsider.com/leopard-preview-ichat-3.jpg http://images.appleinsider.com/leopard-preview-ichat-3.jpg [2] http://www.jeffwright.com/portfolio/portfolio_aol_main.jpg http://www.jeffwright.com/portfolio/portfolio_aol_main.jpg
- jrs235 12y ago
  We need ISPs and the general public to demand that first statement be delivered as "Our customers request an unprecedented amount of data and traffic from Netflix." This is Verizon's problem, not Netflix's. ADD: Netflix doesn't just randomly send data and traffic onto Verizon's network. That data is requested by Verizon ('s customers).
random28345 12y ago
The summary of the article by Verizon is: > Our network is fine. Netflix is buffering because we won't allow the traffic Verizon customers request onto the Verizon network until Netflix pays us more money.
h1fra 12y ago
Can't even load the site, the irony !
josephschmoe 12y ago
Ironically, I can't load the website.
batbomb 12y ago
That chart is worthless without numbers. First off, the chart makes it sound like Netflix is preferring subpar transit providers, and that by simply switching to different transit providers, they'd be fine. But there's no numbers on total bandwidth. For example, if Verizon has 100Gbps to Netflix's content providers, and only 40Gbps to their other content providers, then the additional 20Gbps is going to be a drop in the bucket. It could be the reverse: Netflix is choosing the 40Gbps provider where they could be choosing the 100Gbps, in which case they could easily triple the available bandwidth they have to send to Verizon. But, we all know that's not the real issue. Finally, if peak utilization means that netflix is saturating the router with reduced bitrate video, then increasing that could very well lead to saturation in the internal Verizon network. EDIT: I ran some numbers. If Netflix's providers are saturating at 100%, and other providers are only at 44% peak, and then peak incoming traffic is exiting the Verizon router at 56% peak, that should roughly mean that Netflix has roughly 3.6x the amount of bandwidth from "other transit providers" than it does from Netflix's transit providers, which, in turn, means that Verizon has roughly 2x the total bandwidth that Netflix uses available on it's network. What isn't mentioned is what percentage Netflix uses of it's current providers, how those "other providers" are split up, the maximum theoretical utilization, etc... What isn't mentioned is how that's split up. It's probable Netflix can't use on most of those links even if they wanted to, and there's no word on how many links there are. If it was one giant link, and Netflix could use on it, sure, Netflix should probably try to do that. Bottom line: I don't think there's anything technically limiting Verizon's capacity to Netflix's transit providers, as according to their numbers, Netflix's transit providers only constitute ~22% of the total bandwidth into Verizon. Down the line, it's probable that, even if these link issues were fixed at the border router, they would pop back up again in the Metro Area router or aggregation router, and really gum up Verizon's network.
- deleted 12y ago
  [deleted]
- josho 12y ago
  Great analysis, thank you. Adding to your point about gumming up the metro area routers inside Verizon that may be quite possible. Remember this is Verizon's analysis for a single customer. What are the odds that Verizon, to influence public policy, choose a customer that would best represent Verizon's case. For example, a customer in a region that was just provisioned a bunch of extra capacity. So, it's possible that the percentages at the customer/aggregation/metro area are not typical for Verizon, but rather Verizon's best case scenario.
btoptical 12y ago
If you'd like a different perspective regarding the quality of your broadband connection and the reasons for congestion this is a good read that basically says ATT at least does not provide enough connectivity from the Level 3's optical backbone to their regional networks. I suspect Verizon is probably the same. http://blog.level3.com/global-connectivity/chicken-game-played-child-isps-internet/ http://blog.level3.com/global-connectivity/chicken-game-play... Here's a letter Level3 wrote to ATT on the subject [pdf]. http://www.level3.com/~/media/Assets/legal/cicconi.pdf http://www.level3.com/~/media/Assets/legal/cicconi.pdf
deleted 12y ago
[deleted]
madcow2011 12y ago
The page isn't even loading at all. Just the "his webpage is not available" message. WTG, Verizon.
_nullandnull_ 12y ago
You can always spot a one sided blog post because the ability to leave comments is always turned off. I bet they spent more time building the inforgraphic than they did addressing the customers issue.
- mwfunk 12y ago
  Oh, I don't know about that (although I agree with everyone that Verizon is in the wrong here). They could be announcing free puppies for everyone and the first three comments would be Some Dude Complaining About Obama, a paranoid schizophrenic going on about I don't know what, and somebody claiming that their aunt made $85/hour working from home.
josefresco 12y ago
Seems the issue (according to Verizon) is that Netflix is using "connections specifically to be used only for balanced traffic flows.". Can someone comment on this and the difference between the two (if any)?
trothamel 12y ago
It's always kind of strange to me that Verizon only wants to provide settlement-free peering "where the relative traffic flows between an IP network provider and us remain roughly equal". If you were to take that at face value, it means that Verizon would happy to provide free peering if only Netflix modified its client so it uploaded a number of random bytes equal to the size of the stream it was downloading. But that's nonsensical, right?
- msandford 12y ago
  No, that's fucking GENIUS. Please God, if there is a God, let someone from Netflix read this parent post and implement such a thing to take the wind right out of Verizon's sails.
  - nathanyz 12y ago
    That is actually the basis of our startup is that we can solve the congestion problem by swarming with other video viewers to require less downstream bandwidth usage
    - click170 12y ago
      That's interesting tech, but I kind of feel like it's just postponing dealing with the problem. The problem is anticompetitive behaviour from ISPs, and while this would make video streaming better, it would just allow them to be more abusive, while we try to fill the resulting gap with more technology. IMO we should be leveraging the Netflix situation to fix that behaviour. Regardless, swarm-sourced streaming is something I've been fantasizing about for awhile and I'd love to see your source if it's open. You should definitely proceed with it regardless.
- danielweber 12y ago
  It's a holdover from the old days, when one side of a peering agreement very logically didn't want to carry all the packets of another provider while the other provider doesn't do much to help you in return.
cs702 12y ago
Verizon is blaming Netflix for sending "an unprecedented amount of traffic" and "not taking steps to ensure that there is adequate capacity for their traffic to enter" Verizon's network. The thing is, Netflix is NOT pushing any of that traffic to Verizon. It is Verizon's customers who are pulling that traffic from NetFlix -- and they are paying Verizon to deliver it!!! More here: https://news.ycombinator.com/item?id=7701494 https://news.ycombinator.com/item?id=7701494
- Florin_Andrei 12y ago
  Why are Bridges Dangerous? Dispelling the Killing Myth signed: The Trolls
  - vampirechicken 12y ago
    Is there a special lane for billy goats?
- MrVitaliy 12y ago
  Not to be devils advocate, but if I have 75mbps pipe at home and some internet server is on 1mbps, I'll still be buffering because of the server connection not mine. Right?
  - wdewind 12y ago
    You shouldn't be downvoted for this. It's a reasonable question, it's just not what's happening. The traffic is reaching Verizon (or Comcast etc's) network at reasonable speed, and then congesting.
    - rhino369 12y ago
      But it's also not right to say it's reaching Verizon. It's not. It's just hitting the edge of their network. Now you can say that Verizon is responsible for making sure the edge connection works properly. But you could also say that Netflix shouldn't just dump it's entire load onto shitty congested peer points. Think of Verizon as an island city with 10 bridges. Netflix is trying to send 1/3 of all the traffic through one bridge. This causes congestion on that bridge when the other 10 bridges are moving fine. Verizon is saying, use a different bridge they are all open. Netflix is saying, add more lanes to the one I use because I'm delivering loads to people who ordered it. Both are valid arguments. However, one thing to consider is that Netflix could switch ISPS tomorrow and start dumping their load on a different bridge. Realistically it's absurd that Netflix isn't use a whole variety of ISPs to manage the load themselves. Other big data providers do it. Why can't they? But also, Verizon is probably playing hardball because they are a competitor.
msandford 12y ago
From the article: "Netflix sends out an unprecedented amount of traffic." YES! Yes they do! But Netflix doesn't send that traffic for fun and giggles, just to swamp your network and clog up the port. They do it because YOUR CUSTOMERS HAVE ASKED THEM TO and you, Verizon, you duplicitous shitbag, have SOLD THAT CAPABILITY to your customers. They have LITERALLY PAID you to successfully receive up to 75Mbps worth of traffic from ANYONE, INCLUDING NETFLIX. I can't fucking stand this "they send a lot of traffic our way, but we don't send any back!" bullshit. Netflix doesn't PUSH traffic to your customers, your customers PULL traffic FROM Netflix.
- danielweber 12y ago
  I almost upvoted this.
- Florin_Andrei 12y ago
  Maybe we're getting to the point where Internet connections are more like public roads. Maybe no private entity should control these roads. Maybe it's time for a sweeping change.
  - ProAm 12y ago
    Or just call it a toll road. Of course those tolls will go up every year for things like maintenance and improvements and other esoteric vagaries....
  - genericresponse 12y ago
    I think this might be a clear description of what would happen if roads became privatized. Imagine UPS as Netflix and a turnpike operator as Verizon.
  - ctdonath 12y ago
    The "sweeping change" is an old legal standard: "common carrier". They're contracted to move bytes from source A to customer B at "up to" X Mbps; so long as (albeit limited) bandwidth is available at any given moment, fill the pipe and shove that data thru regardless of source. Traffic shaping may be fair if the pipe is full, but not if unused bandwidth remains sparsely used based solely on source or content.
mutagen 12y ago
Netflix has an open offer to move CDN servers deep into provider networks [1] to avoid peering congestion. This would put most of the Netflix traffic at the bottom right of Verizon's infographic. Why hasn't Verizon accepted this offer? [1] https://www.netflix.com/openconnect https://www.netflix.com/openconnect
- ozymandius182 12y ago
  Not sure why this isn't ranked higher. The entire issue with Verizon is that it refused to do this. They can't complain about intermediate systems when they have been provided the chance to bypass many of them.
- nextweek2 12y ago
  Your comment is the most pragmatic thing I have read. The whole thing of ISP's complaining about Netflix to me is wrong, they are complaining about a peering agreement with a transit provider and just using Netflix as a name people will recognise. The truth is there are technical solutions to these problems but politics is taking over the issue.
xyclos 12y ago
can we all reply to his tweet[1] about this letting him know we can all see through his bullshit [1] https://twitter.com/daedyo/status/487292992504360960 https://twitter.com/daedyo/status/487292992504360960
- swasheck 12y ago
  Nobody's getting responses from either him or Libby Jacobson. They just farted in the room and left.
coldcode 12y ago
They seem to forget that the internet is a request based system. It's not a series of tubes going downhill.
swasheck 12y ago
> We’re Not Satisfied Until Our Customers Are he misspelled "Shareholders"
diminoten 12y ago
The argument I see most commonly from ISPs (and seemingly re-stated here) is basically, "Why should all of our customers pay for the upgrades we'd have to implement if only the Netflix subscribers would benefit from such upgrades?" The idea being that, no matter what they do, the cost of it will just be passed along to the consumer. So their solution is to charge Netflix for the upgrades. That way, when Netflix passes those costs on to the consumers, only the ones who actually have Netflix will be paying for the upgrades, as opposed to all Verizon customers. It makes sense, once you assume Verizon/ISPs and Netflix will just raise their rates to pay off these upgrades/charges. But again, you have to take the assumption that, regardless, the costs will be passed onto the consumers, for granted. If you think the ISPs will try and eat the costs, then this argument doesn't fly. But they're saying they won't, so... they probably won't. I'm not trying to pass moral judgement on this (I gave up trying to convince you guys back when every comment I wrote about this was downvoted to hell and back), just saying what the argument the ISPs are giving actually is. It's just not as "left field" as you might think.
- eyeface 12y ago
  If Verizon feels that Netflix should pass the costs on to the consumers, how would they go about that? Much like Verizon can't know if a user will be using Netflix, Netflix can't know what networks a specific user will come in from, so they'd have to spread the costs over all subscribers, just like Verizon.
  - diminoten 12y ago
    That's a fair point, but Verizon is at least claiming apparently that it's Netflix's problem to solve, not theirs. Maybe varied subscription prices based on ISP. Say, a "base" $5/month you pay to Netflix, and then you can subscribe to "performance" plans that cost an extra $1/$2 per month and get you access to the peering agreements for those ISPs. Though I'm pretty sure that'd start riots. But at least your ISPs wouldn't be offering "fast" lanes! Your products would do that!
jpgvm 12y ago
ITT: People that don't understand that settlement free peering needs to be mutually beneficial. When you pay Verizon you pay for access to their network. Not access to Netflix's network. Verizon will has transit agreements with some people to use it's network and settlement free peering with other ISPs of similar size. When I go to by transit from Verizon to host my website do I expect to get settlement free peering? Well, probably not because I am probably serving much more content than I am consuming, as such my link is unbalanced. If instead I was another ISP and I was sending roughly as much data to Verizon as I am consuming it suddenly becomes mutually beneficial for us to agree to send traffic for free as then we don't each have to pay transit costs for this traffic. Not understanding this is why HN and other tech news outlets are in outrage. This is just how peering arrangements work. Unless your traffic is balanced you need to pay the cost of transit, simple as that. Now, the reason Netflix is bitching is they -are- paying for transit.. they just aren't paying enough to the right people. Their problem is that their transit providers don't have sufficiently large links to Verizons network. Because of the massive asymetical nature of having Netflix on your network Verizon is not agreeing to settlement free peering to Layer 3 et al. Which is completely fair and how things work, unless it's mutually beneficial you have to pay up. Now, why does a witch hunt against those ISPs make no sense? Well simply put they are not running their network at 1:1 contention and probably charge prices that reflect that. If you went and bought carrier grade transit from more respectable providers chances are you would be getting 1:1 contention and these issues would never occur, but you probably pay more than 10x what Netflix is charging per mbit. TLDR: Netflix doesn't understand how telecom works and needs to pay for better transit.
- NotAtWork 12y ago
  Except that I want to pay someone to deliver the traffic I request from third parties (up to a specific rate that I pay for), and just make that work. Not for the privilege to have traffic delivered to me if the sender pays enough money to the government mandated monopoly that owns my area. The value that Verizon gets for accepting that traffic is that I continue to pay them money; no need for the person fulfilling my request to pay Verizon for Verizon to accept things on my behalf (which is what they agreed to do). Verizon (and other ISPs) bill their service as accepting any package on my behalf, and tell me that I can have a certain rate of packages accepted and routed to me. They're simply not upgrading their package receiving centers to reasonably reflect what their customers are ordering because they know that their customers can't do anything about it, and they don't have to provide the service they promised.
freeasinfree 12y ago
Why isn't a large ISP like Verizon directly peered with Netflix? Why waste money and bandwidth on transit?
deeviant 12y ago
If the non-trivial-much-highler-than-netflix monthly cost of verizon's internet service does not cover the cost of verizon connecting their customers the internet, what the hell does it cover? I also don't understand why they are talking about "Review Shows No Congestion on Verizon Network", people are paying them for internet service, not verizon network service.
pedrocr 12y ago
This proves the opposite of what they argue. Supposedly the argument is "Netflix users use too much bandwidth so Netflix should pay for our network upgrades". And yet what their chart and argumentation shows is that the Verizon network has all the capacity it needs to handle this user's traffic and yet at the border of the Verizon network, Verizon is not accepting more traffic from Netflix.
mullingitover 12y ago
Perhaps Netflix would be wise to figure out caching, and let subscribers fully download movies at the highest possible quality with zero buffering. That, or start serving movies with bittorrent.
sandstrom 12y ago
I'm genuinely curious if the author `@daedyo` [twitter] actually believes what he has written? This blog post by level3 was fairly enlightening, and in stark contrast to his view/version of this. http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte... As an side, living in a country where ISPs (mostly) doesn't have monopoly, it's funny that these congestion issues never occur here. Seems like another example where lobbying and money is the underlying issue. https://mayday.us/ https://mayday.us/
Athens 12y ago
The real problem is the ownership of the physical lines that interconnect everything. The high cost and the limited right of way access prevents duplication of physical networks leaving one company in ownership thus control and this is a monopoly. This is why most North American cities (Canada and the United States) have one cable company and one phone company in each region. Services from competitors and small companies just access, use and resell from the single company in the region. In BC for Example you have Shaw and Telus. Bell, Videotron, Rogers rents access over the Telus Network in most cases. What is really needed is to separate the physical network form the service provider. We need a Provincially and State owned network or county or city owned networks with one mandate only. Provide a physical network and charge a price to maintain and grow it. owners of homes and businesses and other buildings would pay for the last mile connection to the network and own that last mile connection. Companies like Bell, Rogers, and dozens upon dozens of small companies offering service be it phone or internet or TV can plug hardware into the network and provide such services. Both the consumer and the ISP pay a network access fee that pays for the network. But this puts every one on equal footing because its the same fee. the physical network can't throttle or control or manipulate any of the content. Would be easy for small companies to expand by adding hardware into data center locations all over the network. Would allow consumers to buy and plug in equipment to allow for point to point VPN connections between family members at a fair price. It is the only way to solve this issue. I would suggest the same be done with the Cell Phone networks. The Cities and Provinces and States own the towers using all the frequencies and they charge a access fee to the subscriber and phone provider to manage and expand the network. Any company could start up a cell service and compete head to head. As long as the core infrastructure is owned by one entity that has its own motives on how to use it, it will always be a monopoly situation and thats bad for business and consumers and innovation. Take the expensive part out of the puzzle and make it a public owned resource let the ISP and companies fight for customers on equal footing.

106 points by danielsamuels 12y ago

Verizon's customers request an unprecedented amount of data and traffic from Netflix. This is Verizon's problem, not Netflix's. Verizon should be working with Netflix on an amicable solution to perhaps cache data inside Verizon's network... O What do you know! Netflix already has a program for that: https://www.netflix.com/openconnect https://www.netflix.com/openconnect

diminoten 12y ago
To be fair, this problem is non-existant today, because Netflix has entered into payed peering agreements with Verison/Comcast/other ISPs. They more or less acquiesced. They're just not happy about it.

rnovak 12y ago

They can claim whatever they want, but when you tunnel Netflix traffic through a VPN so all the ISP see's is encrypted traffic and bandwidth issues disappear, it removes all mystery. I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish. considering what we're paying for broadband, we should be getting what we pay for without resorting to obfuscating our data.

kevinskii 12y ago
I tried the same thing a couple of years ago and observed the same, and then switched providers.
- cordite 12y ago
  If only more had that option.
  - pcl 12y ago
    In SF, WebPass (and MonkeyBrains, from what I hear) are great alternative ISPs. Both do point-to-point wireless. My building shares a symmetric 100Mbps connection to WebPass, and I pretty reliably see 30-40Mbps on my wired network. And I pay significantly less than I had previously paid Comcast. (However, I never used the TV component of my Comcast offer, so the cost structure is presumably different for people who do want cable TV.) I understand that point-to-point wireless is more realistic in an urban environment than in the suburbs or a more rural setting. Do other cities have good alternate ISPs, or are SF residents somewhat unique in their appetite to try alternate providers?
    - sparkman55 12y ago
      Point-to-Point (Fixed) Wireless is actually also an attractive solution in rural / semi-rural settings as well. If the distances are moderate (with line-of-sight) but demand is too sparse to warrant the cost of running coaxial / fiber (and distances are too far for DSL), a common alternative on the high end is fixed microwave, and on the low end Verizon LTE. A good example is the Ames, Iowa area, where an ISP has set up a tower with good line of sight, and a bunch of directional antennas both on the tower and at customers locations. This setup provides great service out to ~15 miles or so, enough to cover the outskirts of this small town. Some issues with this setup: storms cause 'rain fade' which blocks the signal, and the wireless spectrum does not scale to too many users. Also, there's a finicky antenna on the roof that can become misaligned in storms.
    - bicknergseng 12y ago
      My building in SF is locked in to ATT Uverse. I'd kill for WP or MB.
  - Someone1234 12y ago
    That's really the key problem here. There is lack of choice in the US and Canada. Everyone loves to drum their chest about capitalism but in this case it isn't working, and ISPs have clearly colluded to divvy up the territories so they aren't directly competing. The internet has become a public utility just like electricity, water, or the roads. Businesses and private citizens depend on it day to day, and economic growth is hampered by poor performance in this area (e.g. new types of businesses that depend on bandwidth, like a Netflix competitor, cannot exist if the pipes are slow). In my view the raw cable and fibre should be confiscated from the big three by the government using eminent domain, and then rented out to competing ISPs (including the big three) on even footing. The rent would pay to maintain the cables/fibre and deploy more network, and the ISPs could add a premium before reselling it to end users to cover support costs and maintaining equipment (e.g. routers, edge connections, etc). Basically make the final mile public property. It would make real competition as smaller ISPs can now compete with large ones, and there isn't regularly problems for these companies (e.g. getting permits to place new cables).
- seanflyon 12y ago
  Assuming your new provider did not have the problem, which ISP did you switch to?
random28345 12y ago
> I've tried several VPN's to test out my theory, and with each one, as soon as the ISP couldn't inspect traffic, all throttling seem to magically vanish. That's not inconsistent with Verizon's statement. Verizon is saying, "we won't accept more traffic directly from Netflix (via their transit providers) until we get more money!". By using a VPN, you're avoiding that direct connection, and traversing a less-loaded link.
- swasheck 12y ago
  Exactly. They took Netflix's bait with the "congestion" message. They're demonstrating that their network isn't congested and now we're closer to seeing that they really are attempting to charge multiple customers for the same traffic.
agwa 12y ago
That's because your VPN providers are no doubt using a different transit provider than Netflix (Level 3) so your traffic is bypassing the congested peering points. The peering disputes are affecting not just Netflix traffic, but traffic between Verizon and any site using Level 3 for transit. (Not to take the blame away from Verizon. I'd love to see them try to use their logic to explain why other, non-Netflix services work like crap from Verizion.)
- joeframbach 12y ago
  Isn't the Netflix-to-Verizon transit something that we already pay Verizon to maintain?
  - maxsilver 12y ago
    Yes, but Verizon also wants Netflix to pay them for that same transit. That's what the whole blog article is about. Verizon trying to invent valid reasons for why they want Netflix to pay them.
    - jpgvm 12y ago
      They are not inventing reasons. Netflix is serving the content, Netflix pays for it to move. Simple as that. The issue is Layer 3 is not buying enough transit from Verizon to match the disparity in the traffic it wants to send beyond what is covered by settlement free peering. If you look at the diagram on the Verizon post you can see the link is saturated in one direction and only at 30% utilization in the other direction. This is the DEFINITION of transit. The fact Netflix is even blaming Verizon for this is ridiculous.
      - Terretta 12y ago
        > If you look at the diagram on the Verizon post The diagram is wrong. The thing they colored red is not red. The wire is fine. The box inside Verizon's network has a network card in it that's oversubscribed. A slice of their box should be red. Btw, the definition I'm familiar with for "transit" is crossing a network backbone, it's neither serving, nor consuming. The traffic "transits" your network, on its way from somewhere to somewhere.
azurelogic 12y ago
I had this same experience with Comcast. See also Level 3's blog post calling Verizon and other ISPs out on this BS: http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte...
- tempestn 12y ago
  Exactly. Verizon can try to spin this as "Netflix is pushing this traffic on us through inferior intermediate links," but what's really happening is that Verizon's customers are requesting the traffic. The standard response when one of their border links is congested is to work with the peer to upgrade it. Level3 is willing to do so. Verizon would rather try to squeeze money out of Netflix for an unnecessary "direct link". Hopefully people see through this PR.
- nathanyz 12y ago
  Yep this is exactly the fast lane that we don't want. The lane is not any faster, just less traffic, says Verizon--really, that seems like a fast lane to me.
  - deleted 12y ago
    [deleted]
  - cmiles74 12y ago
    Since Verizon is eliminating so much traffic, those FIOS links are free for downloading all kinds of content their customers don't want.
digitalpacman 12y ago
I want Verizon to reply exactly to this more than I want them to reply to anything Netflix has said.
kyberias 12y ago
Assuming you connected your Verizon IP to a VPN server that resides in a non-Verizon network, that test is insufficient to deduce that the ISP looks at the data. As Verizon is saying, Netflix does not pay enough for the data that they send. They don't need to inspect the data. By connecting your VPN to a non-Verizon network, Verizon sees the Netflix stream coming from that network (not Netflix).
- dasil003 12y ago
  How does Verizon know if Netflix is paying enough for the data they send? They are not Netflix's ISP.
gabriel34 12y ago
Couldn't it be that your VPN provider has better/less crowded connections to both Verizon and Netflix? I can see how using a lower grade link to a main source of traffic outside a network can actually shape the traffic and reduce the workload on said network. If that is not Verizon's doing and if there is some reason why Verizon is not connected to other Netflix's links then I can see it as being out of Verizon hands. It is more probable, though that the limitations in interconnection between Verizon and Netflix are intentional so as to avoid high utilization of Verizon's network.

sergiotapia 12y ago

It's so strange, I live in Bolivia and Netflix never buffers for me at all. I have a laughable 200kbps real download speed (1768kbps plan) and I only wait for Netflix to load for about 2 seconds on my PS4 and off I go watching on 480SD quality. I would expect the US to be much much better.

skelsey 12y ago
No one in the US wants to watch 480 video.
- maxsilver 12y ago
  Or can. My 5mbps AT&T DSL struggles to load Hulu and Netflix on a daily basis, even at just 480p
- lostcolony 12y ago
  I do. Comcast. Data cap. Of course, what I would -prefer- is to ditch Comcast, and yes, watch at a higher resolution, and I will at the first available opportunity. But can't. Because monopoly.
  - swasheck 12y ago
    nonsense. there is no monopoly in capitalism. /sarcasm
    - adventured 12y ago
      If there was Capitalism you'd actually be able to compete.
      - swasheck 12y ago
        There is Capitalism ... but it's been corrupted. Economic systems are morally neutral. How people employ them determines the morality. The U.S. is, in general, pushing Capitalism to its logical end, in which the system consumes itself in the name of its constituents' "best interests."

jflowers45 12y ago

Let the verbal battle begin!

wavefunction 12y ago

I know who I believe, and it's not Verizon!!!

dcc1 12y ago

LOL taking 30 seconds+ to even load their page

jrs235 12y ago
They must be using Netflix's network.
RandomSpamMan 12y ago
Doesn't even load for me. http://i.imgur.com/wT7zV3f.png http://i.imgur.com/wT7zV3f.png
josefresco 12y ago
Yeah the site/page has been down/slow. Getting ISE 500 errors.
eddieh 12y ago
The first time the page failed to load, the second time it took 52.76 seconds according to the Chrome Developer Tools. I'm on a university network that is super fast.

esoterae 12y ago

How apropos that the link is showing unavailable after timeout & errors after only 17 minutes of being posted. <Verizon> It's definitely not us. Yep. Not it.

idiot900 12y ago

A well written PR piece, but no new insights here. Verizon feels it is not Verizon's job as an Internet service provider to provide adequate service to parts of the Internet that its customers are paying them to access. This wouldn't be a problem if broadband access were actually a competitive market - customers would simply switch to a provider that had taken the initiative to give its customers what they want.

seanflyon 12y ago
Thank you. While almost all the comments are saying the same thing, I think you stated it best.

angersock 12y ago

Somewhat ironically, the page is taking forever to load.

cdnsteve 12y ago
500 Internal Server Error for me right now

JimmaDaRustla 12y ago

Why is the "transit providers" split into two segments - inbound and outbound? Wouldn't it be the same logical segment and be impacted based on data in both directions?

random28345 12y ago
> Why is the "transit providers" split into two segments - inbound and outbound? Because they are trying to illustrate how unbalanced the traffic flow is in each direction, to bolster their argument that Verizon shouldn't have to deliver the streams that Verizon customers requested until Netflix gives them more money.

LandoCalrissian 12y ago

Long story short, Level 3 won't pay us for peering agreements so it's their fault.

techwizrd 12y ago
The most incredible part is that the facts and anecdotes presented so far don't agree with Verizon's tale, so instead of Verizon decided to respond with a fluffy PR piece instead of fixing the problem or countering with facts. Verizon seems completely unwilling to take any blame in this and does not seem to have their customer's best interest at heart. There are no new facts in this article other than a useless infographic and repeated claims that Netflix is saturating their link with content Verizon customers are asking for.
- yaur 12y ago
  The only new fact I see here is that link between L3 and Verizon is not symmetric. That's not surprising really, but it means that there is already traffic asymmetry baked into whatever peering agreement they have today.
fryguy 12y ago
It seems like Netflix is the one providing value to the Verizon network so it should be the other way around. It would be like someone saying "I'm not downloading these files from a webserver because they're not paying me to download it."

pizza 12y ago

Can anyone recommend networking resources analogous to Learn C the Hard Way or Dive into Python?

thrush 12y ago
High Performance Browser Networking by Ilya Grigorik is pretty solid, and free online. http://chimera.labs.oreilly.com/books/1230000000545 http://chimera.labs.oreilly.com/books/1230000000545
bqe 12y ago
Beej's Guide to Network Programming is still the best. http://beej.us/guide/bgnet/ http://beej.us/guide/bgnet/

malchow 12y ago

"Netflix sends out an unprecedented amount of traffic" is one of several misleading locutions in this piece. Verizon customers request these packets. The ISPs' problem is that they aren't willing to change their pitch to customers. As long as their product is "internet service at ~x gbps," they are on the wrong side of this issue. So they go ransoming the content folks. The funny thing is that the dream world of the big modern ISPs would appear to be something like AOL circa 1994, where you can have "the internet," but it's a cheap lo-fi interconnect [1], and what they mainly want you to consume is a bunch of low-quality in-house content channels. [2] [1] http://images.appleinsider.com/leopard-preview-ichat-3.jpg http://images.appleinsider.com/leopard-preview-ichat-3.jpg [2] http://www.jeffwright.com/portfolio/portfolio_aol_main.jpg http://www.jeffwright.com/portfolio/portfolio_aol_main.jpg

jrs235 12y ago
We need ISPs and the general public to demand that first statement be delivered as "Our customers request an unprecedented amount of data and traffic from Netflix." This is Verizon's problem, not Netflix's. ADD: Netflix doesn't just randomly send data and traffic onto Verizon's network. That data is requested by Verizon ('s customers).

random28345 12y ago

The summary of the article by Verizon is: > Our network is fine. Netflix is buffering because we won't allow the traffic Verizon customers request onto the Verizon network until Netflix pays us more money.

h1fra 12y ago

Can't even load the site, the irony !

josephschmoe 12y ago

Ironically, I can't load the website.

batbomb 12y ago

That chart is worthless without numbers. First off, the chart makes it sound like Netflix is preferring subpar transit providers, and that by simply switching to different transit providers, they'd be fine. But there's no numbers on total bandwidth. For example, if Verizon has 100Gbps to Netflix's content providers, and only 40Gbps to their other content providers, then the additional 20Gbps is going to be a drop in the bucket. It could be the reverse: Netflix is choosing the 40Gbps provider where they could be choosing the 100Gbps, in which case they could easily triple the available bandwidth they have to send to Verizon. But, we all know that's not the real issue. Finally, if peak utilization means that netflix is saturating the router with reduced bitrate video, then increasing that could very well lead to saturation in the internal Verizon network. EDIT: I ran some numbers. If Netflix's providers are saturating at 100%, and other providers are only at 44% peak, and then peak incoming traffic is exiting the Verizon router at 56% peak, that should roughly mean that Netflix has roughly 3.6x the amount of bandwidth from "other transit providers" than it does from Netflix's transit providers, which, in turn, means that Verizon has roughly 2x the total bandwidth that Netflix uses available on it's network. What isn't mentioned is what percentage Netflix uses of it's current providers, how those "other providers" are split up, the maximum theoretical utilization, etc... What isn't mentioned is how that's split up. It's probable Netflix can't use on most of those links even if they wanted to, and there's no word on how many links there are. If it was one giant link, and Netflix could use on it, sure, Netflix should probably try to do that. Bottom line: I don't think there's anything technically limiting Verizon's capacity to Netflix's transit providers, as according to their numbers, Netflix's transit providers only constitute ~22% of the total bandwidth into Verizon. Down the line, it's probable that, even if these link issues were fixed at the border router, they would pop back up again in the Metro Area router or aggregation router, and really gum up Verizon's network.

deleted 12y ago
[deleted]
josho 12y ago
Great analysis, thank you. Adding to your point about gumming up the metro area routers inside Verizon that may be quite possible. Remember this is Verizon's analysis for a single customer. What are the odds that Verizon, to influence public policy, choose a customer that would best represent Verizon's case. For example, a customer in a region that was just provisioned a bunch of extra capacity. So, it's possible that the percentages at the customer/aggregation/metro area are not typical for Verizon, but rather Verizon's best case scenario.

btoptical 12y ago

If you'd like a different perspective regarding the quality of your broadband connection and the reasons for congestion this is a good read that basically says ATT at least does not provide enough connectivity from the Level 3's optical backbone to their regional networks. I suspect Verizon is probably the same. http://blog.level3.com/global-connectivity/chicken-game-played-child-isps-internet/ http://blog.level3.com/global-connectivity/chicken-game-play... Here's a letter Level3 wrote to ATT on the subject [pdf]. http://www.level3.com/~/media/Assets/legal/cicconi.pdf http://www.level3.com/~/media/Assets/legal/cicconi.pdf

deleted 12y ago

[deleted]

madcow2011 12y ago

The page isn't even loading at all. Just the "his webpage is not available" message. WTG, Verizon.

_nullandnull_ 12y ago

You can always spot a one sided blog post because the ability to leave comments is always turned off. I bet they spent more time building the inforgraphic than they did addressing the customers issue.

mwfunk 12y ago
Oh, I don't know about that (although I agree with everyone that Verizon is in the wrong here). They could be announcing free puppies for everyone and the first three comments would be Some Dude Complaining About Obama, a paranoid schizophrenic going on about I don't know what, and somebody claiming that their aunt made $85/hour working from home.

josefresco 12y ago

Seems the issue (according to Verizon) is that Netflix is using "connections specifically to be used only for balanced traffic flows.". Can someone comment on this and the difference between the two (if any)?

trothamel 12y ago

It's always kind of strange to me that Verizon only wants to provide settlement-free peering "where the relative traffic flows between an IP network provider and us remain roughly equal". If you were to take that at face value, it means that Verizon would happy to provide free peering if only Netflix modified its client so it uploaded a number of random bytes equal to the size of the stream it was downloading. But that's nonsensical, right?

msandford 12y ago
No, that's fucking GENIUS. Please God, if there is a God, let someone from Netflix read this parent post and implement such a thing to take the wind right out of Verizon's sails.
- nathanyz 12y ago
  That is actually the basis of our startup is that we can solve the congestion problem by swarming with other video viewers to require less downstream bandwidth usage
  - click170 12y ago
    That's interesting tech, but I kind of feel like it's just postponing dealing with the problem. The problem is anticompetitive behaviour from ISPs, and while this would make video streaming better, it would just allow them to be more abusive, while we try to fill the resulting gap with more technology. IMO we should be leveraging the Netflix situation to fix that behaviour. Regardless, swarm-sourced streaming is something I've been fantasizing about for awhile and I'd love to see your source if it's open. You should definitely proceed with it regardless.
danielweber 12y ago
It's a holdover from the old days, when one side of a peering agreement very logically didn't want to carry all the packets of another provider while the other provider doesn't do much to help you in return.

cs702 12y ago

Verizon is blaming Netflix for sending "an unprecedented amount of traffic" and "not taking steps to ensure that there is adequate capacity for their traffic to enter" Verizon's network. The thing is, Netflix is NOT pushing any of that traffic to Verizon. It is Verizon's customers who are pulling that traffic from NetFlix -- and they are paying Verizon to deliver it!!! More here: https://news.ycombinator.com/item?id=7701494 https://news.ycombinator.com/item?id=7701494

Florin_Andrei 12y ago
Why are Bridges Dangerous? Dispelling the Killing Myth signed: The Trolls
- vampirechicken 12y ago
  Is there a special lane for billy goats?
MrVitaliy 12y ago
Not to be devils advocate, but if I have 75mbps pipe at home and some internet server is on 1mbps, I'll still be buffering because of the server connection not mine. Right?
- wdewind 12y ago
  You shouldn't be downvoted for this. It's a reasonable question, it's just not what's happening. The traffic is reaching Verizon (or Comcast etc's) network at reasonable speed, and then congesting.
  - rhino369 12y ago
    But it's also not right to say it's reaching Verizon. It's not. It's just hitting the edge of their network. Now you can say that Verizon is responsible for making sure the edge connection works properly. But you could also say that Netflix shouldn't just dump it's entire load onto shitty congested peer points. Think of Verizon as an island city with 10 bridges. Netflix is trying to send 1/3 of all the traffic through one bridge. This causes congestion on that bridge when the other 10 bridges are moving fine. Verizon is saying, use a different bridge they are all open. Netflix is saying, add more lanes to the one I use because I'm delivering loads to people who ordered it. Both are valid arguments. However, one thing to consider is that Netflix could switch ISPS tomorrow and start dumping their load on a different bridge. Realistically it's absurd that Netflix isn't use a whole variety of ISPs to manage the load themselves. Other big data providers do it. Why can't they? But also, Verizon is probably playing hardball because they are a competitor.

msandford 12y ago

From the article: "Netflix sends out an unprecedented amount of traffic." YES! Yes they do! But Netflix doesn't send that traffic for fun and giggles, just to swamp your network and clog up the port. They do it because YOUR CUSTOMERS HAVE ASKED THEM TO and you, Verizon, you duplicitous shitbag, have SOLD THAT CAPABILITY to your customers. They have LITERALLY PAID you to successfully receive up to 75Mbps worth of traffic from ANYONE, INCLUDING NETFLIX. I can't fucking stand this "they send a lot of traffic our way, but we don't send any back!" bullshit. Netflix doesn't PUSH traffic to your customers, your customers PULL traffic FROM Netflix.

danielweber 12y ago
I almost upvoted this.
Florin_Andrei 12y ago
Maybe we're getting to the point where Internet connections are more like public roads. Maybe no private entity should control these roads. Maybe it's time for a sweeping change.
- ProAm 12y ago
  Or just call it a toll road. Of course those tolls will go up every year for things like maintenance and improvements and other esoteric vagaries....
- genericresponse 12y ago
  I think this might be a clear description of what would happen if roads became privatized. Imagine UPS as Netflix and a turnpike operator as Verizon.
- ctdonath 12y ago
  The "sweeping change" is an old legal standard: "common carrier". They're contracted to move bytes from source A to customer B at "up to" X Mbps; so long as (albeit limited) bandwidth is available at any given moment, fill the pipe and shove that data thru regardless of source. Traffic shaping may be fair if the pipe is full, but not if unused bandwidth remains sparsely used based solely on source or content.

mutagen 12y ago

Netflix has an open offer to move CDN servers deep into provider networks [1] to avoid peering congestion. This would put most of the Netflix traffic at the bottom right of Verizon's infographic. Why hasn't Verizon accepted this offer? [1] https://www.netflix.com/openconnect https://www.netflix.com/openconnect

ozymandius182 12y ago
Not sure why this isn't ranked higher. The entire issue with Verizon is that it refused to do this. They can't complain about intermediate systems when they have been provided the chance to bypass many of them.
nextweek2 12y ago
Your comment is the most pragmatic thing I have read. The whole thing of ISP's complaining about Netflix to me is wrong, they are complaining about a peering agreement with a transit provider and just using Netflix as a name people will recognise. The truth is there are technical solutions to these problems but politics is taking over the issue.

xyclos 12y ago

can we all reply to his tweet[1] about this letting him know we can all see through his bullshit [1] https://twitter.com/daedyo/status/487292992504360960 https://twitter.com/daedyo/status/487292992504360960

swasheck 12y ago
Nobody's getting responses from either him or Libby Jacobson. They just farted in the room and left.

coldcode 12y ago

They seem to forget that the internet is a request based system. It's not a series of tubes going downhill.

swasheck 12y ago

> We’re Not Satisfied Until Our Customers Are he misspelled "Shareholders"

diminoten 12y ago

The argument I see most commonly from ISPs (and seemingly re-stated here) is basically, "Why should all of our customers pay for the upgrades we'd have to implement if only the Netflix subscribers would benefit from such upgrades?" The idea being that, no matter what they do, the cost of it will just be passed along to the consumer. So their solution is to charge Netflix for the upgrades. That way, when Netflix passes those costs on to the consumers, only the ones who actually have Netflix will be paying for the upgrades, as opposed to all Verizon customers. It makes sense, once you assume Verizon/ISPs and Netflix will just raise their rates to pay off these upgrades/charges. But again, you have to take the assumption that, regardless, the costs will be passed onto the consumers, for granted. If you think the ISPs will try and eat the costs, then this argument doesn't fly. But they're saying they won't, so... they probably won't. I'm not trying to pass moral judgement on this (I gave up trying to convince you guys back when every comment I wrote about this was downvoted to hell and back), just saying what the argument the ISPs are giving actually is. It's just not as "left field" as you might think.

eyeface 12y ago
If Verizon feels that Netflix should pass the costs on to the consumers, how would they go about that? Much like Verizon can't know if a user will be using Netflix, Netflix can't know what networks a specific user will come in from, so they'd have to spread the costs over all subscribers, just like Verizon.
- diminoten 12y ago
  That's a fair point, but Verizon is at least claiming apparently that it's Netflix's problem to solve, not theirs. Maybe varied subscription prices based on ISP. Say, a "base" $5/month you pay to Netflix, and then you can subscribe to "performance" plans that cost an extra $1/$2 per month and get you access to the peering agreements for those ISPs. Though I'm pretty sure that'd start riots. But at least your ISPs wouldn't be offering "fast" lanes! Your products would do that!

jpgvm 12y ago

ITT: People that don't understand that settlement free peering needs to be mutually beneficial. When you pay Verizon you pay for access to their network. Not access to Netflix's network. Verizon will has transit agreements with some people to use it's network and settlement free peering with other ISPs of similar size. When I go to by transit from Verizon to host my website do I expect to get settlement free peering? Well, probably not because I am probably serving much more content than I am consuming, as such my link is unbalanced. If instead I was another ISP and I was sending roughly as much data to Verizon as I am consuming it suddenly becomes mutually beneficial for us to agree to send traffic for free as then we don't each have to pay transit costs for this traffic. Not understanding this is why HN and other tech news outlets are in outrage. This is just how peering arrangements work. Unless your traffic is balanced you need to pay the cost of transit, simple as that. Now, the reason Netflix is bitching is they -are- paying for transit.. they just aren't paying enough to the right people. Their problem is that their transit providers don't have sufficiently large links to Verizons network. Because of the massive asymetical nature of having Netflix on your network Verizon is not agreeing to settlement free peering to Layer 3 et al. Which is completely fair and how things work, unless it's mutually beneficial you have to pay up. Now, why does a witch hunt against those ISPs make no sense? Well simply put they are not running their network at 1:1 contention and probably charge prices that reflect that. If you went and bought carrier grade transit from more respectable providers chances are you would be getting 1:1 contention and these issues would never occur, but you probably pay more than 10x what Netflix is charging per mbit. TLDR: Netflix doesn't understand how telecom works and needs to pay for better transit.

NotAtWork 12y ago
Except that I want to pay someone to deliver the traffic I request from third parties (up to a specific rate that I pay for), and just make that work. Not for the privilege to have traffic delivered to me if the sender pays enough money to the government mandated monopoly that owns my area. The value that Verizon gets for accepting that traffic is that I continue to pay them money; no need for the person fulfilling my request to pay Verizon for Verizon to accept things on my behalf (which is what they agreed to do). Verizon (and other ISPs) bill their service as accepting any package on my behalf, and tell me that I can have a certain rate of packages accepted and routed to me. They're simply not upgrading their package receiving centers to reasonably reflect what their customers are ordering because they know that their customers can't do anything about it, and they don't have to provide the service they promised.

freeasinfree 12y ago

Why isn't a large ISP like Verizon directly peered with Netflix? Why waste money and bandwidth on transit?

deeviant 12y ago

If the non-trivial-much-highler-than-netflix monthly cost of verizon's internet service does not cover the cost of verizon connecting their customers the internet, what the hell does it cover? I also don't understand why they are talking about "Review Shows No Congestion on Verizon Network", people are paying them for internet service, not verizon network service.

pedrocr 12y ago

This proves the opposite of what they argue. Supposedly the argument is "Netflix users use too much bandwidth so Netflix should pay for our network upgrades". And yet what their chart and argumentation shows is that the Verizon network has all the capacity it needs to handle this user's traffic and yet at the border of the Verizon network, Verizon is not accepting more traffic from Netflix.

mullingitover 12y ago

Perhaps Netflix would be wise to figure out caching, and let subscribers fully download movies at the highest possible quality with zero buffering. That, or start serving movies with bittorrent.

sandstrom 12y ago

I'm genuinely curious if the author `@daedyo` [twitter] actually believes what he has written? This blog post by level3 was fairly enlightening, and in stark contrast to his view/version of this. http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte... As an side, living in a country where ISPs (mostly) doesn't have monopoly, it's funny that these congestion issues never occur here. Seems like another example where lobbying and money is the underlying issue. https://mayday.us/ https://mayday.us/

Athens 12y ago

The real problem is the ownership of the physical lines that interconnect everything. The high cost and the limited right of way access prevents duplication of physical networks leaving one company in ownership thus control and this is a monopoly. This is why most North American cities (Canada and the United States) have one cable company and one phone company in each region. Services from competitors and small companies just access, use and resell from the single company in the region. In BC for Example you have Shaw and Telus. Bell, Videotron, Rogers rents access over the Telus Network in most cases. What is really needed is to separate the physical network form the service provider. We need a Provincially and State owned network or county or city owned networks with one mandate only. Provide a physical network and charge a price to maintain and grow it. owners of homes and businesses and other buildings would pay for the last mile connection to the network and own that last mile connection. Companies like Bell, Rogers, and dozens upon dozens of small companies offering service be it phone or internet or TV can plug hardware into the network and provide such services. Both the consumer and the ISP pay a network access fee that pays for the network. But this puts every one on equal footing because its the same fee. the physical network can't throttle or control or manipulate any of the content. Would be easy for small companies to expand by adding hardware into data center locations all over the network. Would allow consumers to buy and plug in equipment to allow for point to point VPN connections between family members at a fair price. It is the only way to solve this issue. I would suggest the same be done with the Cell Phone networks. The Cities and Provinces and States own the towers using all the frequencies and they charge a access fee to the subscriber and phone provider to manage and expand the network. Any company could start up a cell service and compete head to head. As long as the core infrastructure is owned by one entity that has its own motives on how to use it, it will always be a monopoly situation and thats bad for business and consumers and innovation. Take the expensive part out of the puzzle and make it a public owned resource let the ISP and companies fight for customers on equal footing.