Peering agreements existed in the form they did because of the underlying assumption that traffic would be roughly symmetric. Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.
This is one of the underreported issues in the whole net neutrality debate. In the late '90s I worked in the WAN engineering group doing capacity planning at a big regional CLEC, which actually was a Tier 1 backbone provider. It was widely assumed in the industry that the flat rate pricing model -- pay $X for Y "guaranteed" bandwidth -- was living on borrowed time and would be replaced by actual usage-based pricing, paying for packets like you pay for gallons of water. The flat rate model only worked because we generally could keep available capacity well ahead of demand. We could do that because we were adding customers like crazy, and customers sucking down high bandwidth were outliers. If those prerequisites ever changed we'd be screwed.
Well, here we are in 2014 and ISPs like Comcast are not adding customers like crazy anymore, because most of the people who want high-speed data from them already have it. But the customers they do have are, on a per capita basis, using much more bandwidth now than they were five years ago. And this is only going to get worse. The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever, but those requirements are no longer being driven by customer acquisition. Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.
But wait, doesn't Netflix pay for all their bandwidth already? Well, probably not: unless industry practices have changed drastically, even those backbone trunks are actually oversubscribed. If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. Netflix breaks that assumption.
I don't know quite what the solution here is, and I certainly don't want the kind of dystopian "the ISP only gives you access to web sites that you pay for" future the sharpest critics paint. But I don't see this as a step toward that dystopian future. What I see it as is the ISPs trying to figure out how to work usage into pricing models without saying "screw it" and actually adopting a usage-based pricing model.
If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7.
Whoa, whoa, whoa. This is exactly the assumption commercial customers paying for unmetered links are making, otherwise they wouldn't be paying for unmetered links. Not being oversubscribed is supposed to the entire point. If ISPs' business models depend on them being able to break their promises to customers paying for continuously saturated links that's their problem, but they're still on the hook for what they're being paid to provide.
I'm looking forward to chipotle_coyote response on this, but I don't think he's referring to the circuits that commercial customers pay for, but instead the ISP peering circuits - which are usually expected to run at much less utilization.
Isn't flat rate pricing completely orthogonal to net neutrality? Net neutrality doesn't mean you can't charge me for usage, it just means you can't charge me differently based on the kind of usage, as opposed to the volume.
I personally think the resistance to usage-based pricing in the tech community is really weird, although I mostly see it on the consumer end. Hard caps are bad, and caps that only apply to some services are really bad, but charging more if you use more is completely reasonable.
The idea is that usage-based pricing doesn't make sense for customers because there is a hard limit on consumption simply based on the terrible state of last mile connections. Simply put, charging 2014 dollars for usage on a line with a capacity that was last relevant in 1985 is not motivated by actual capacity limitations, but a simple money grab.
(Not to mention that there are no inherent variable costs to transporting bits. Water needs to be purified, electricity is obviously usage priced - but you can be pushing bits forever once you have the hardware setup.)
I think the resistance to usage-based billing is practical rather than economic. With a flat-fee scheme you can look at your bill and say, "yup, $89.95 is what I agreed to pay each month."
With usage-based billing, the amount will float up and down and it's hard to understand how the amount is calculated. Usage-based telephone bills are broken down by calls, which you can probably remember.("What's this 46 minute call to 555-1234? Oh yeah, I called Mom last week.")
With IP traffic, how do you itemize the charges? Temporally? I used to get cell phone bills that included pages and pages of stuff like "Jan 27, 4pm, 239789987345 bytes". It was completely useless.
How about reverse DNS? That has other problems.
• You use BitTorrent? Here's 247 hosts with incomprehensible names on your bill.
• You use a VPN? A whole bunch of traffic is lumped together.
• "Honey, what's all this traffic to nastyporn.com?"
• It puts pressure on web sites to optimize their deployment strategy for billing clarity
The "correct" way is probably to log everything, then provide analytics software. But I can't imagine my Mom generating reports just to understand her internet bill.
Economically, sure, usage-based billing is a great idea. Alignment of incentives and all that. But practically, it's a pain for everybody involved.
The nightmare scenario is basically upon us: capacity requirements are climbing faster than ever [...] Which means either the ISPs eat the cost of upgrading their infrastructure (ha!) -- or they raise rates.
I dunno.. being profitable enough to buy time-warner cable for billions doesn't sound like a nightmare scenario from their POV. It sounds like they'd rather spend the money becoming a monopoly than improve their infrastructure.
You forgot to mention the part about how it actually does balance out because Comcast is receiving $18B billion a year for internet bandwidth from the its customers. Customers are requesting more data, but Comcast wants to be paid on both sides because they have a monopoly over the customer.
If it is really expensive to accept Netflix's offer of free hardware and free bandwidth then the additional cost should be passed to the customer. The problem is that it's not actually more expensive and Comcast knows their customers can't choose someone else so it's a much better strategy to squeeze the content providers.
Comcast is prepared to provide that bandwidth to its customers subject to it's fair use cap of 300 Gigabytes. What they aren't prepared to do (for better or worse) is install larger circuits to Cogent so that Cogent's customers have unblocked access to Comcast's customers.
Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast. This will probably set a very bad precedent for them when it comes to renew those agreements in 5-10 years, or however long they will be. Maybe they are hoping for some competition to enter the US environment by then.
> Netflix could have held out, and made Comcast out to be the bad guy, or could have leaned on Cogent to get the circuits installed - but, in this case, they folded and just decided to pay Comcast.
How would that work? Comcast customers by and large can't switch to another broadband provider and Netflix customers can certainly switch to another video provider (Comcast!). It's not like this is a brand new issue, Netflix has held out for a long time. The Comcast / TWC merger is what finally did it, 30M combined customers is too big to risk on a public game of net neutrality chicken.
Surely they're still working on lobbying efforts to make it illegal for a company like Comcast to simultaneously have a monopoly on selling broadband and compete with services using said broadband. It's pretty obvious at this point that internet access is a utility and it should be treated as such.
"If you pay for (say) a DS3 line, there's an implicit assumption that you are not going to be pumping 44.736 Mbit/s through it 24/7. "
This may be the cases for ISP-ISP connections, but every transit circuit I've ever purchased as an end-user has two elements, a Non Reccurring Charge (NRC) to set up the connection, and then a monthly recurring charge (MRC) consisting of, a port speed, and a 95th percentile metered usage charge.
e.g. I might get a gigabit port set up for an up front $5000 NRC, and then a 24 month contract for $2000/month port speed, and then pay $5/megabit @95th percentile of 5 minute averages sampled over a month.
The ISP samples the usage of my port every 5 minutes, (8640 samples in a 30 day month), drops the top 5% (432 samples), and then charges me the cost of the next sample (sample #8208) - If the sampled speed at #8208 was 1 Gigabit, and sample #8207 was 0 kilobits, then I am still charged $2000/month + 1000 * $5 = $7000/MRC for that month.
Cogent has always had a reputation in the industry for being the cut-rate player (to the point of other ISPs just dropping them as peering connections, resulting in companies literally not being able to communicate with their customers) - Netflix was trying to rely on Cogent to send data to their customers, presumably because they were charging significantly less than "Tier-1" transit-providers, and finally came to the realization that yes, you do get what you pay for.
None of this surprises me, and none of it appears relevant to the conversation regarding Net-Neutrality - this is a peering discussion, pure and simple.
You forgot to mention that network technology has leapfrogged last mile connections many many times over. In many ways, the problems have gotten considerably easier for many ISPs, and there are no capacity problems at big exchanges, and certainly no barriers to scaling up.
> Services like Netflix that send massive amounts of traffic in a single direction violate this assumption.
Netflix does not send any data. Comcast customers request lots of data from Netflix. Significant difference.
Another example (just in the other direction): A backup provider needs to pay up big-cable because their customers are uploading a lot of data?
Your second example is actually really fascinating - backup and photo providers get awesome deals on bandwidth because that part of the circuit is typically highly underutilized. So, ironically, it's a case where backup providers argue that their circuits should be charged at much less than market rates because the traffic is going in the opposite direction ,and their provider usually has circuits going idle in that direction.
I'm willing to wager that backup providers haven't run into any upload issues with Comcast for this reason.
DS3 line? What is this, 1996?
Each individual Comcast customer has in theory paid for a DS3 worth of bandwidth (modulo the "cap"), which should be a reality-check that some of your assumptions from that era need updating.
The defense (and it was always a thin one) of ratio clauses in peering agreements was hot potato routing. The ingress traffic is what you actually have to haul a long distance, so the agreement says "you must meet us everywhere we peer, and you must maintain ratio." Together, those rules mean each individual path will be asymmetric but the overall load on the WAN will be shared between the peers. It was always slightly silly, though.
Hot potato is now over. CDN's are desperate to do all the work and bring traffic as close to eyeballs as the monopolists will allow because the cost of actually building a network is irrelevant compared to the rent the monopolists are extracting. The monopolists are selling so-called "transit" to haul packets 1 mile. They're really charging for access to their captive eyeballs, not for moving bits. If you want to send a few megabits to Asia, fine, whatever. You can do it for the same price. They're not even mad, "glad we could make a deal," etc. The situation is nakedly broken and warped, and everybody actually in the business knows it.
Peering has always been a mix of market power, shaming, and political excuses ("ratio" was one). What we're now seeing is the shaming and excuses fall away thanks to apologist bloggers and weak politicians, and market power emerge as the only thing that matters. Compared to this power, traditional outdated notions of neutrality are irrelevant: there is no need to give traffic less "preferential" treatment when you can simply depeer them and then try to sell them so-called "transit" to go 1 mile. "Paid peering" == "well, how much money you got? Let's make a special deal just for you".
I used DS3 as an example. But no, even in 2014, you've only paid for a DS3 (or whatever) worth of bandwidth in theory. It's not just the last mile that's oversubscribed -- so are all the trunks leading up to it. If that's changed between then and now, I would lay pretty good money that it's changed to make things worse, not better.
Without meaning to express either approval or cynicism, market power has always been the only thing that matters in this business. But CDNs have paid big ISPs special rates for years, because they really do put special strains on infrastructure. And my impression remains that that's what this particular battle is ultimately about.
you are some 5 yrs late... ISPs already put usage in the contracts.