11 ms·
Clever technique to draw attention away from what is most likely a contractual dispute. Verizon's argument is clear: > Netflix chose to attempt to deliver tha
by sheetjs 12y ago
Clever technique to draw attention away from what is most likely a contractual dispute.
Verizon's argument is clear:
> Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows
Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
- chimeracoder 12y ago> (in this case, since Level3 is pushing more data, they would be the ones paying). That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers. All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern. Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).
- sliverstorm 12y agoit would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this) Not impossible though. I'd bet money that equal in/out contracts would be cheaper- maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts?
- chimeracoder 12y ago> maybe they just hoped no one would notice, or didn't anticipate the impact of Netflix on already-existing contracts? It's not just Netflix - the entire Internet works this way. End-users send a request, and servers send a response. There are very few cases in which the response is not larger than the request. It doesn't matter what content you're serving - the raw traffic delivered to consumers is going to be greater than the raw traffic from them[0]. Saying that they didn't know that the traffic would be shaped this way is saying that Level 3 doesn't understand the way the Internet functions on a technical level, which I would say is impossible. [0] This is (sort of) why residential Internet speeds are usually quoted asymmetrically (e.g. 20 down/5 up). Consumers - well - consume more traffic than they produce, so networks are already optimized for delivering more traffic to consumers than they themselves generate. That said, all of the traffic is initiated by consumers; the traffic Netflix sense is only in response to the explicit request by a paying Verizon customer.
- mikecb 12y agoI guess the idea is that backbone providers will each have a similar number of ISPs (and content networks) to deliver traffic from and to. That's clearly not the case, especially with Verizon providing both a backbone and an ISP, and level 3 providing mostly content delivery. Maybe both should work on diversifying the clients that they serve.
- sliverstorm 12y agoYou seem to have segmented ISPs in your head. As if there were "Hosting" ISPs and "Client" ISPs. Level 3 provides service to customers of both varieties. You can't jsut look at Level 3 and say, "Well they are a Hosting ISP so they send more than they receive".
- regularfry 12y agoISPs are loosely segmented into "Hosting" and "Client", except they're usually called "content" and "eyeball".
- 12y ago
- msandford 12y ago> in this case, since Level3 is pushing more data, they would be the ones paying No. That's not how the internet works. Nobody PUSHES data. People PULL data. Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data. They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase. If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network. But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it. This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.
- angersock 12y agoWell put. Yours is probably the clearest explanation I've seen of why this all feels so icky.
- moultano 12y agoAs I like to say it, saying that Neflix uses a lot of bandwidth is like saying that the Hetch Hetchy uses a lot of water pipes.
- sheetjs 12y agoYou still haven't refuted the point. This isn't about the relationship between Verizon and its customers, at all. The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon. Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.
- 12y ago
- epistasis 12y ago>Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows Verizon's argument would be a lot stronger if they didn't attempt to say that Netflix chose balanced flow contracts. They are saying that Netflix went shopping for backbone providers and chose the ones that had balanced traffic flow agreements to deliver video across, which is just silly. Such rhetorical overreaching is a common debating error, and can ruin an otherwise good argument. Verizon may or may not have balanced flow agreements, I don't know what the contracts are, but now I don't trust Verizon to honestly report on them based on their imputation of Netflix's motivations.
- chiph 12y agoThis is the exact argument that Verizon is using -- that Level3 is sending "too much" data. Of course, that data is in response to requests sent from Verizon's residential and business customers. So it's not unwanted -- in fact it's the data that their customers are paying Verizon to deliver to them.
- smsm42 12y agoI'm not sure why Level3 has to pay anything. Let's imagine there's only 4 entities in the world - User, Verizon, Level3 and Netflix. User buys internet access from Verizon and pays. Netflix hosts servers at Level 3. Level 3 has connection to Verizon. User pays Verizon to be able to access Netflix servers. Netflix pays Level 3 to deliver content up to Verizon. Why when the data crosses between Level3 and Verizon Level3 should pay anything? Verizon has been already paid by the user to deliver the data. That's like a taxi driver demanding payment from the restaurant since he brings them clients. Clients already paid for being brought to the restaurant, that's the whole reason they hired taxi in the first place, and demanding payment from the restaurant sounds like a racket to me. And a fraud towards the users which expect the provider to perform their obligations, not to use them as bargaining chip to extort other businesses.
- danielweber 12y agoIf there were only 4 entities, your description would make perfect sense. It's a waterfall model, where the data only falls from Netflix to Level3 to Verizon to User. But the Internet is a big hierarchy where things flow downhill. Every node can talk with every other node. Historically the networks would trade data freely if they were both doing an equal amount, but even 15 years ago I heard sober network engineers worrying that in the future there might be very unbalanced connections and what would be the fair way to deal with that. NB: In the end I think Verizon is being purposefully difficult to their customers, but they are not completely unreasonable to think that having a lot of data sent into their networks [1] is just something they should STFU about. [1] yes, even if you say "data requested by their customers" over and over again
- smsm42 12y agoThe shouldn't shut up about data sent to their networks. They should get paid for it. And they already are! The only reason for consumers to buy services from Verizon (I mean private consumers not running server farms) is to have data sent through Verizon networks and get the data to the users, and of course it is wildly asymmetric - everybody knows it is. It's not some externality that Verizon must suffer - it's the primary and sole reason why users subscribe to Verizon services and pay them money! If Verizon is unable to provide this service for the money they change, they should charge more or change how they do it, but what I completely don't get is how it is taken as if their direct contractual obligations - taking data from whenever it is and getting it into my home (provided I am a Verizon customer) - is presented as something extra that Verizon must unexpectedly do for free.
- cortesoft 12y agoBalance-flow contracts are only used when each network is acting as a gateway where the end-point of the traffic is NOT IN either of the parties involved in the contract. For example, suppose there are 5 separate networks, run by different groups: A, B, C, D, and E. C connects to A,B and D, and D connects to C and E. In this case, the only way networks A and B can connect to D or E is through the connection between C and D. Now, if in this arrangement, the D or E networks are sending/requesting a lot of traffic to/from networks A and B, while C is rarely requesting data from D or E, then network C might have a fair argument to make that D is abusing their balanced-flow peering agreement. Basically, they are being asked to be a middle-man for all the traffic going between networks that are NOT theirs. They might ask for compensation for doing this job (carrying traffic between two other networks). However, this is NOT what Verizon does. All of the traffic between level 3 and the Verizon network is bound FOR A VERIZON CUSTOMER. They are not a middle-man, they are the end destination. Now, if Level-3 is sending data over those congested level-3/verizon gateways that is bound for, say, AT&T's customers, then they would have an issue and could fairly demand payment. That is not the case, however.
- neura 12y ago"That is not the case, however." is indeed correct. The case is that Verizon does indeed act as a peer for business connections. Verizon is not just a consumer based end point on the internet. This is the crux of it all. Verizon is charging customers for access to internet content and they're bringing that content in over a peer balanced connection with agreements in place, intended for use in an actual peering environment, by business costumers of Verizon who are paying to provide content in the same way that Netflix pays Level3 to provide their content over their peer connections. The conflict here is that Netflix is paying to provide the content, Verizon's consumers are paying to get access to the content. That creates a connection chain that is unbalanced in some way. Verizon just wants that imbalance to be double payment for them and nobody else.
- malchow 12y ago@sheetjs: I'm sorry, but I simply cannot follow the logic in the Verizon argument that you quote. You quote: "Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows" But isn't that a misstatement of fact? CDNs like Level3 do nothing but transfer data, usually media files, onto consumer ISP networks when those ISP's customers' request them. In what dream world are Akamai and Level3 receiving as much data from Verizon's consumer network as they are sending? Why would any ISP expect a link with a CDN to have a balanced traffic flow?
- wmf 12y agoLevel 3 is not just a CDN; it's a major backbone. Likewise Verizon is not just DSL & FIOS because Verizon Business is also a major backbone. Apparently Level 3 and Verizon did have balanced traffic for many years.
- malchow 12y ago@wmf, OK, I did not know that background. But I can easily imagine that the AWS infrastructure is geographically diverse enough that Netflix is using Level3 only as a CDN, not as an NTP.
- wmf 12y agoThis dispute isn't just about Netflix (although they precipitated it); all traffic between Level 3 and Verizon is being affected.
- deleted 12y ago[deleted]
- kenrikm 12y agoThe Verizon users are REQUESTING the data, they pay Verizon to provide it, it's Verizon's job to make sure the users have access to the "Internet" Verizon looks at it as if Netflix and Level 3 are pushing the Data, everyone else clearly sees that they are pulling it.
- kevin_thibedeau 12y agoIt doesn't matter if there is an imbalance in the traffic flow. Verizon is being paid to deliver requested packets over IP. That's all they have to do. Singling out Netflix is pointless. If Netflix were replaced with 1000 video streaming sites with the same aggregate traffic volume who would they blame for all the data their customers are pulling in? Verizon desperately wants to avoid being a humble ISP with no way to pull the strings the way do with their telco service. That's why they're playing this game.
- pdonis 12y ago> Clever technique to draw attention away from what is most likely a contractual dispute. Oh, please. If that's really what's stopping this from being fixed, then I would be hard pressed to find a better illustration of the need for regulating "last mile" providers as public utilities. We're talking about an investment that is not even rounding error in Verizon's accounting, to connect Level 3's network to a Verizon network that, by Verizon's own admission, has plenty of excess capacity that is not being used. So the whole idea that, if the traffic isn't balanced, Verizon is somehow incurring huge extra costs that they need to be compensated for, is obviously bogus, which means their contractual terms are nothing less than an attempt to use their privileged position to extract huge monopoly rents in exchange for no added value whatsoever.
- jwecker 12y agoIn my (admittedly limited) experience with peering I've never seen a contract. Lots of ad-hoc verbal agreements, and occasionally emails outlining a mutual understanding, but because it so often is obviously beneficial to both parties it is almost always handled in a casual way and decisions are deferred to network engineers for engineering considerations. I've never seen legal get involved. If Verizon brought it to a point where it is a contract dispute then it would be just as much of an indictment- it doesn't change the substance of the post at all.
- einrealist 12y agoThat would be the case when Level3 would like to peer with Verizon on order to send traffic to another peering party's WAN of Verizon. But it is Verizon customers, who request the traffic and Verizon is not providing the best quality, because they don't want to upgrade their own equipment and the peering with e.g. Level3. In fact, it should be Verizon paying to Level3.
- Yver 12y agoThere's too much confusion, you're right. What Netflix need is some hard numbers. I suggest they take very detailed statistics at the client level and constantly report them back to their servers, in real time. With all that data available, it will surely ~~artificially balance the network usage and solve the legal aspect~~ allow both parties to make informed decisions.