20 ms·

Level3 is without peer, now what to do?

781 points by mortimerwax 12y ago

brokenparser 12y ago
This wouldn't happen if those ISPs didn't have local monopolies. Networks should be opened by selling traffic wholesale to other companies so that they can compete for subscribers on those networks. The network owners would have more than enough money for upgrades and if they don't, downlevel ISPs will sue them.
- kasey_junk 12y ago
  I'm not making a claim about the veracity of the counter argument to this, but it is easy to state. If network providers had to sell traffic wholesale to other companies, they would have less incentive to upgrade them as those networks then become pure commodities. Right now the local monopolies can use their networks to sell high margin services (cable) bundled with low margin services (internet traffic). If they had to compete with other companies, namely bargain ISP offerings, the cost of ownership of the network would be non-profitable. I've heard some argue that this is precisely why DSL has lagged so far behind cable in upgrades.
  - rlpb 12y ago
    > they would have less incentive to upgrade them as those networks then become pure commodities Pit DSL against cable, as has happened in the UK. The POTS network is owned by one company (BT) and the cable network by others. Retail customers generally have a choice of connecting to the Internet by either. BT (POTS network owner) are required to sell traffic wholesale to competitor ISPs (who then buy their own transit). This seems to work very well here, and DSL upgrades continue.
    - kasey_junk 12y ago
      That describes almost precisely how the US system works.
      - opendais 12y ago
        [text removed since it seem to have confused the person who replied] http://news.cnet.com/FCC-changes-DSL-classification/2100-1034_3-5820713.html http://news.cnet.com/FCC-changes-DSL-classification/2100-103... "The ruling puts phone companies on the same regulatory footing as cable companies, which are exempt from having to offer access on their infrastructure to competing Internet service providers."
        kasey_junk 12y ago
        I'm not an expert, but I'm reasonably certain that Cable providers in the UK are also not required to provide ISP access via their networks.
        rlpb 12y ago
        No, they are not. Otherwise I'd be using one.
        opendais 12y ago
        Yes but he is claiming the US system works like the British system...when it doesn't.
    - robk 12y ago
      British DSL is far worse than cable in my experience. Virgin's cable & fiber offering is massively faster and has a wider footprint than DSL products I've seen.
      - rlpb 12y ago
        > British DSL is far worse than cable in my experience. Define "worse". What ISP were you using? From a peering point of view as described in the article, there are a large number of ISPs buying transit and then selling Internet connections through BT-provided last mile DSL lines. I never suffer from peering congestion, since my (DSL) ISP pride themselves on not having any. For the link from my local exchange to my ISP, there are multiple options, too, and my ISP monitors congestion on my line closely. I currently have no congestion there, either, and if I did, there are multiple providers available (thanks to LLU): http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-purpose.html http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-purpose.ht... The only issue with DSL in the UK is the (mostly analogue) quality of the DSL line itself, and the way that BT (the company with the monopoly on POTS lines) manages them. And perhaps the pace of upgrades (eg. to fibre to a street cabinet and copper from there, instead of copper all the way from the exchange), but upgrade rollouts are happening (and fibre to the cabinet is already available to me).
    - 7952 12y ago
      TV is also a huge factor in the UK. Both BT and Sky need high broadband speeds to be able to offer video on demand that can compete with virgin.
  - jbooth 12y ago
    The article said that their cable business is way less profitable than their internet business. I haven't done any research on the topic, but given that they have to pay content providers for cable, and don't for internet, I'm inclined to believe Cringely -- even though in a normal market it should be a low-margin business, selling undifferentiated bits for $.
    - danielweber 12y ago
      If it was "selling bits for $" we would already be done. But today's ISP capacity is unsuitable for tomorrow's ISP demands. One might hand-wave the network as "sunk costs" but the ISP has to keep on sinking those costs. In fact, that's what Crinkley is complaining about: the ISP isn't sinking costs into its peer with L3.
      - jbooth 12y ago
        Sure, so you've got depreciating capital costs on a schedule coupled with ongoing operational costs, that gets you to some amount of $/yr to operate the network, and $/year/bit in terms of network capacity. What should be happening is competition driving down the price per bit to some amount just marginally above their costs. These costs may be high but the margins should be low. That's Econ 101 and it's what everyone who ever advocated for the free market is making a case for. What appears to be happening is they're collecting a big fat surplus on top of that figure, and that surplus is in addition to all the wasted money in their corporations that's never had to be cut in a down quarter.
  - opendais 12y ago
    And those people are completely 100% wrong. DSL got screwed for around 10 years by having more expensive regulatory costs than Cable. Why yes, if you tax X more than Y, Y is going to have the advantage.
- yk 12y ago
  We have essentially such a system in Germany, does not work well. Thing is, that the owner of the network has then to compete with the resellers, but the resellers need to play nice with the owner. The result is a total mess of regulation to force the network owner ( T-Systems) to play nice.
  - a3n 12y ago
    Back in the dial-up days I had my choice of ISPs. Some people chose on price, I chose on quality/service, and mine wasn't that much more than the cheapos. I can't recall any issues with the underlying phone company operator of the wires, although that may be because the wires were primarily for phone and they have to work by law (I think). I miss the days of being able to choose my ISP. Now I have Comcast, with CenturyLink DSL an inferior possible second choice.
  - NorthGuy1 12y ago
    We have the same system in Canada and the wholesaler ISP (Bell) famously imposed unpopular usage based billing on the re-seller (TekSavvy) to kill the only advantage it had over the ISP. http://www.cbc.ca/news/technology/crtc-approves-usage-based-internet-billing-1.879213 http://www.cbc.ca/news/technology/crtc-approves-usage-based-...
JoshTriplett 12y ago
Somebody has to pay the money to upgrade the equipment and bandwidth available at these exchange points. The very reasonable argument in this article is that the ISPs should pay that cost, which seems reasonable given that their customers are demanding it. It sounds like the ISPs are playing a game of chicken, trying to see if their peers like Level3 will throw money in to pay for the ISP to upgrade its equipment and bandwidth. That's certainly something the ISPs can try to do; on the other hand, what are their customers going to do, not use Netflix and YouTube? If a pile of customers of one ISP start reporting that they're all having a poor experience with high-bandwidth video, and there are a pile of well-publicized press releases blaming the ISP, customers will start complaining to the ISP, and they'll have to upgrade their infrastructure eventually. (And in areas where they have competition, there's an incentive to upgrade before the competitors, to avoid losing customers; while there isn't such competition in every locale, there are enough locales with more than one ISP choice to make those customers painful to lose.) But what does any of that have to do with mandated peering requirements at the NSFnet exchanges? Who would enforce that, and why, when any two major networks can set up peering at any number of meet-me rooms? Requiring that an ISP peer as much traffic as is available or not peer any at all seems ridiculous; some ISPs will suck more than others, but that's the problem of them and their customers, not a problem for the entire Internet. Meanwhile, I'm surprised there aren't more startups and VCs looking to bet that "new ISP that doesn't suck" is a viable business model. People are chomping at the bit for Google Fiber, which seems unlikely to grow to a national level without developing competitors. This is a space with very few competitors, and there hasn't been serious competition in that space since DSL stopped being a viable option.
- bsimpson 12y ago
  Google's ability to print money in conjunction with its generally high PR ratings give it the cache to take on a project like fiber. I can't imagine it's cheap or easy to lay fiber if you don't have substantial financial backing as well as a political goal you're trying to achieve. That said, I've used both Sonic.net and MonkeyBrains here in San Francisco. Never had to rely on a major ISP in this town, and I like it that way.
  - 12y ago
xhrpost 12y ago
I like the article overall but I don't understand the author's proposed solution. The issue as it stands is apparently a lack of peering, in that big ISPs are using transit to reach large content providers rather than directly peering to those networks. So how would "kicking them out" for a maxed out connection work? If I buy transit from Level3 and my connection maxes out, I'm no longer allowed to be a customer of Level3?
- logfromblammo 12y ago
  I think the proposal is to simply sever the link when the peering connection saturates. The traffic out through the link from LazyISP is generating more traffic in through the link than the pipe can handle. LazyISP pretends that this is not their problem, that the Internet is somehow trying to shove its traffic through poor ol' LazyISP's pipes. The claim is analogous to traffic exiting a Autobahn-style limited-access highway and attempting to reach its destination faster by taking the narrow country backroads to get to the on-ramp of the next highway. The only real reason for any traffic to route to LazyISP is if one of the endpoints is in there, and there is seldom a reason for that traffic to go anywhere but the shortest route between the end point and the nearest backbone highway access point. The typical argument from LazyISP is so ridiculous on its face that it makes me furious every time I see it. All of those guys present asymmetric upload and download speed to residential customers. Every last one of them. And they pretend that when all of their customers' asymmetric bandwidths are summed up, the total is somehow magically supposed to be symmetric. Every business on the planet whose cash flow depends on pushing bits out to the Internet goes directly to the backbone with content delivery networks, or at least to an ISP that is serious about peering. They are not going to an ISP with crappy interconnects. There is no way in hell that any cul-de-sac on the network map like a last-mile ISP is ever going to push as many bits to the backbone network as it pulls from it, especially when they do nasty tricks like throttling, port blocking, and hidden metering. The point is that a steaming pile like Time Warner does not have any internal content providers that any of its customers want to connect to, precisely because of its business practices. If Time Warner cannot connect to the Internet, it has nothing to offer the customer through that precious last mile connection except television channels. So they stop paying the $25-150 a month and try to make do with standing on a ridge waving semaphore flags at each other. LazyISP needs Level 3 more than Level 3 needs them. Therefore, when LazyISP tries to extort them, the correct response should be, "Fuck you," as they flip the off switch.
rrggrr 12y ago
Godaddy, Rackspace, Google, Amazon, etc. have skin in this game. With multiple redundant network connections they could, for a day or a week, defend neutrality by shaping their traffic to the lowest common denominator or routing their traffic to avoid the peer's punitive bottlenecks. Today its Level 3 and Netflix, but tomorrow it could easily be them.
signet 12y ago
If a customer is paying for an internet connection, they are paying for access the full internet, to the best of their ISP's abilities. This is the net neutrality law we need: ISPs should be compelled to upgrade their backbone links as they become congested, to satisfy their customer's demand. Congestion can be easily monitored and often these peerings are "free". (Yes there is a non-zero cost to increase switch and router capacity and to have someone plug the cables in, but it's not like Level3 is charging for the bits exchanged.) But the point is, since most ISPs are de-facto monopolies in this country, we need rules telling them they have to upgrade their capacity to meet their customers demand, if they are promising broadband speeds.
- siculars 12y ago
  This. ISP's of the like mentioned by the OP are being paid for INTERNET access. Not access to the local network resources on the ISP's network. If I pay for an X up/Y down connection to the internet then that is what I expect. I expect X up/Y down to any host/content on the internet (given that host can reply at that speed, etc.)
- mattmcknight 12y ago
  We probably need pricing more directly based on bandwidth consumed, maybe with peak hour pricing, as opposed to just based on "speed" with high bandwidth caps.
  - morgante 12y ago
    Don't buy into the cable PR. They have incredibly high margins and could easily make upgrades without further gouging customers.
    - rgbrenner 12y ago
      "incredibly high"? Comcast and Time Warner have profit margins of 8-10%. For comparison, Target has a profit margin of 3%, and Google's is 21%. Sounds more mid-range to me. As for your second point, that they could easily make upgrade without additional charges.. have any numbers backing that up? The last mile is expensive.
deleted 12y ago
[deleted]
spindritf 12y ago
Except it’s actually right (not wrong) because those bits are only coming because customers of the ISPs — you and me, the folks who have already paid for every one of those bits — are the ones who want them. What is the source of the notion that, because you paid for your consumer broadband, all bits are paid for and the charge for carrying them cannot be split with the other side of the connection? Why is it so bizarre that both sides of the connection have to pay for it? Because you're used to your phone working differently? As an analogy, you know how you used to pay for a subscription to a magazine and there were ads inside which advertisers (the other side of the connection via the magazine in this case) also paid for? The magazine split its fee in two: you paid part of it, and the advertisers paid the other part. It's the same here. There is nothing fundamentally wrong with charging both sides. You may prefer a different fee structure but a better argument than "I already paid for it!" is necessary.
- wging 12y ago
  In this case, the difference is that broadband subscribers are not being sold a magazine which they know will contain ads--they are being sold a full connection to the internet! That's the precedent, that's what we think we're getting for our money.
  - spindritf 12y ago
    What does full connection to the Internet mean? No ISP can guarantee it because any other independent network can drop traffic at will. And the Internet is just a bunch of independent networks. The service you believe you paid for doesn't and cannot exist. I can filter out all Comcast IPs on my firewall tomorrow if I feel like it. Many people actually do it to third world providers.
    - talmand 12y ago
      There's a difference between bits not being delivered because of the originating server failing to deliver and my ISP actively preventing delivery. "Unlimited Internet" has already been established in sales pitches for years, the language defined once people understood the concept of bandwidth caps. "Unlimited" means you have unlimited "access", not unlimited "bandwidth". This has been the industry's very own sales pitch for years.
api 12y ago
While I agree with the general thrust of the article, there is one fallacious argument here. Cringely argues that cable breaks even and money is made on the net, but that's an artificial distinction. What if cable disappeared? Would they still make money if they had to pay for the upkeep of the network with only Internet fees? The desperation and risk of this game of chicken convinces me that the answer might be "not much." The loss of cable might very well be apocalyptic for these companies, at least from a shareholder value and quarterly growth point of view. What's happening is very clear to me: the ISPs are trying to either harm the Internet to defend cable or collect tolls on streaming to attempt to replace cable revenue. That's because cable is dying a slow death. This is all about saving cable. The fundamental problem is that cable ISPs have an economic conflict of interest. They are horse equipment vendors that got into the gas station business, but now the car is driving out the horse and their bread and butter is at stake.
- Jtsummers 12y ago
  > Would they still make money if they had to pay for the upkeep of the network with only Internet fees? In parts of the country, slower-speed copper, fast-download cable, and a few fiber networks are already built out. The cable distribution giants like Time Warner Cable and Comcast are already making a 97 percent margin on their “almost comically profitable” Internet services, according to Craig Moffet, an analyst at the Wall Street firm Bernstein Research. As Levin points out, “If you are making that kind of margin, it’s hard to improve it.” And most Americans have no choice but to deal with their local cable company. -- http://www.technologyreview.com/news/510176/when-will-the-rest-of-us-get-google-fiber/ http://www.technologyreview.com/news/510176/when-will-the-re... If that's accurate, I suspect that they could live without cable as we know it today and still make plenty of money. Working on finding out more numbers to see what the revenue breakdown is between cable and internet, though I wouldn't mind if someone else chimed in. EDIT: Found this [1] for Comcast. For 2012 and 2013 they've lost video customers, increased internet customers. Had growth in revenue for both categories, but a greater increase from internet (percentage and absolute). So the way things are heading, cable leaving would cost them a lot. However, they can probably make up for it elsewhere, and, ultimately, should be planning on cable TV going the way of the dodo regardless of the nearterm revenues and profits. It's not a viable longterm strategy, like SMS (while immensely profitable) for cell carriers, a good short term strategy but as content shifts to the generic data channels the specialized channels will fall into disuse. [1] http://files.shareholder.com/downloads/CMCSA/3182774232x0xS1193125-14-47522/1166691/filing.pdf http://files.shareholder.com/downloads/CMCSA/3182774232x0xS1... EDIT 2: Check out page 53 of the linked filing. Video revenue for 2013 was $20.5 billion. Programming costs were $9.1 billion. So the net, ignoring any other costs, right there is $10.4 billion. If the ISP side of the house is actually making 97% margins like the claim in the Technology Review article, then it is making almost as much as the video side of the house. At the current growth rates it'll surpass the video profit, but not revenue, sometime in the next few years. Their NBCUniversal acquisition may reduce their programming costs, I'm not sure of the legalities involved in that (their growth and acquisitions could put them afoul of anti-trust/monopoly laws and regulations).
sbierwagen 12y ago
Regulate ISPs as utilities.
- hga 12y ago
  Common carriers are the words of art for these sorts of utilities. And Comcast and company are in steadily increasing danger of this as they piss off more and more voters (at a certain point, all the money in the world is of no interest to a politician who perceives himself in danger of having to spend more time with his family after the next election; gun owners have demonstrated this many times).
lifeisstillgood 12y ago
1. Peering is based on equal traffic both ways. At the moment we tend to download gigabytes with a few bytes of request. As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise 2. rise of ad-hoc local networks This might come out of mobiles, this might be me dreaming, and it might come with sensible home router designs, but ultimately most of the traffic I care about probably originates within 2 miles of my house - my kids school, traffic news, friends etc A local network based on video comms - that will never happen. just like mobile phones. 3. electricity and investment In the end this is down to government investment. Let's not kid ourselves, gas, water, electricity, railroads, once they passed some threshold of nice to have into competitive disadvantage not to have, governments step in with either the cash or the big sticks. Fibre to the home, massive investment in software engineering as a form of literacy, these are the keys to the infrastructure of the 21C and it's a must have for the big economies, and it's a force multiplier for the small.
- dragonwriter 12y ago
  > As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise Aside from the chicken and egg problem, I don't think you ever get to a point where that really is the case. Not because video comms never take off, but because even when it does its a small share of content delivery and the net is still wildly asymmetrical.
- ep103 12y ago
  The problem is your point #3 has never been true, though. In each case, in American history (though I don't know about water), the utility was privatized for _years_, became local monopolies, and used that monopolistic advantage to become one of the most major Washington lobbying industries of the time, often to the massive detriment to specific sections of the economy. In each case it was only _after_ an alternative industry rose in lobbying prominence in washington (often, the next one on your list) that the previous utility lost their position as #1 lobbying firm in Washington, that their competitors were able to then force anti-trust lawsuits, and or get fair government regulation. The railroads were monopolized until the late 1800s, the original American idea of a fat cat comes from railroad owners who made profits at the expense of midwestern farmers. That was eventually regulated when oil became a major lobbying industry, whereupon oil and gas subsidies became the new norm and we regulated the railroads. Electricity is a general complex area, but outside of domestic coal-based electricity production, my understanding is that the majority(?) of American electricity comes from nuclear power, which in turn is a regulated monopoly. That industry is so locked that Northern America imports much of its electricity from nuclear plants in Canada, and Texas is actually on Mexico's electricity grid. I was somewhat hopeful that as Wallstreet rose to replace other industries as one of the top lobbying firms in America that they would end up pushing for net neutrality so as to commoditize their network costs, but instead they simply pushed for B2B fiber to be regulated completely differently (and more sanely) than consumer networks. I don't want to wait for something better than the internet to come around, before we get net neutrality back.
mhandley 12y ago
Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours. Would certainly cost Comcast a fair amount in customer support calls, bad publicity, and properly bring the debate to the general public.
- klodolph 12y ago
  It would also give the CDNs a fair amount in customer support calls, bad publicity, and make their paying customers angry about lost business.
  - ep103 12y ago
    I don't know about bad publicity though. To each customer support call simply say, Comcast has broken their contract with our CDN. Please call comcast. Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin, people will easily believe that, especially since in this case, it would be true. Level3 et all are enabling this behavior.
    - chimeracoder 12y ago
      > Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin And Time Warner is the second: http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-survey-study/ http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-... Nobody likes the cable companies. It would be very easy for someone to capitalize on that hatred to win public support for net neutrality; it's just a matter of crafting the right campaign.
      - coldpie 12y ago
        This is why I'm so disappointed to see the small-medium ISPs hop right in bed with Comcast and co-sign anti-net neutrality letters: http://arstechnica.com/business/2014/05/top-isps-threaten-to-be-less-innovative-and-spend-less-on-network-upgrades/ http://arstechnica.com/business/2014/05/top-isps-threaten-to... With the national attention this issue is getting, this is a tremendous marketing opportunity to take customers away from the big dogs.
keehun 12y ago
Maybe I'm naïve beyond any recognition, but shouldn't the ISP's or whoever is peering charge based on the bandwidth amounts? It sounds like they have a flat-rate contract with each other and now they're charging more?
- awor 12y ago
  From the article, it seems like the "flat-rate contract" the ISPs have for peering with L3 is $0. The issue that L3 is raising, is that the equipment/appliances that the ISP has installed in these peering arrangements are insufficient to keep up with the amount of traffic which the ISP's customers are requesting, and L3 is attempting to provide. If the ISPs were to upgrade their peering equipment to handle their customer's requests, there would be no issue.
  - keehun 12y ago
    So by Netflix making a deal with Comcast, did L3 effectively get cut out of the picture? Also, isn't increasing equipment/bandwidth part of the business? It seems so ridiculously absurd that Comcast is balking to increasing its bandwidth...
- hga 12y ago
  Inexperienced in the field? This sort of thing has been going on since at least the '90s; this search ought to bring up some relevant hits: https://www.google.com/search?q=uunet+peering+ma-east https://www.google.com/search?q=uunet+peering+ma-east One thing to focus on is that there are 2 customers, and two middlemen. E.g. Netflix <-> Level 3 <-> Comcast <-> a Netflix subscriber. Netflix and the Netflix subscriber are each paying their ISPs hefty sums for access to "the Internet". In the ideal world, Level 3 and Comcast would peer at various points for maximum efficiency on both their networks. Comcast enjoys a local municipality government monopoly in their markets, with iffy competition from state and Federal monopolies granted to the RBOCs Verizon, AT&T, and in theory CenturyLink (which owns the US West states, but has never been very serious about Internet service and especially their own "TV"/cable service), and all the small ILECs who provide telephone service where the big boys had no interest. One big problem here is that "TV"/cable service deployment for the first two RBOCs, who cover the vast majority of the population, has been halted outside of contractual requirements with municipalities and the like. (If you're in a really bad spot, like I am, on unincorporated land adjacent to two cities, the local cable company is not required to provide service and doesn't, AT&T provides slow and expensive (but reliable!) ADSL with very small caps, 150 GiB per month ... and now they're buying the DirectTV my parents use to get around that cable company problem!) So as noted in the article, the bandwidth flow might be asymmetrical, and for these local "last mile" ISPs, there are technical and economic reasons for only providing asymmetrical service in the first place, e.g. head end noise, but all the Netflix bits coming in through Level 3 to Comcast Netflix subscribers are paid for, by the hefty payments the subscribers pay to Comcast for access to "the Internet". (And which are soon going to get caps as well from what I've read.) But Comcast would of course prefer you get your "TV"/cable bits from them, and have no particular interest in enabling Netflix to supplant them. And they're now committing fraud on their subscribers, by telling them they're providing access to "the Internet", but deliberately degrading significant portions of it.
pessimizer 12y ago
This cable-menu style image from the comments is scary: http://i.huffpost.com/gen/1567010/original.jpg http://i.huffpost.com/gen/1567010/original.jpg
- qnaal 12y ago
  Fearmongering- the internet is not going to be that easy to kill.
  - pessimizer 12y ago
    If they stop supplying it to us without discrimination it will. The internet is made up of plastic and metal boxes that are under other people's control - not magic.
    - sliverstorm 12y ago
      Things that cannot be simply cut off: - populist revolutions Things that can: - electricity - the internet
- wmf 12y ago
  But the clever thing about Comcast's behavior is that the customer does not see anything like that image.
droopybuns 12y ago
So on the one side is the fat-cat ISP who doesn't want to make expensive capital investments ih their transport. And on the other side is the fat-cat vc funded video content providers, who don't want to pay for the their mp4-based saturation of all the pipes. This is a negotiation. There are two active media campaigns that are trying to gin up our anger against The Other Guy (tm) as part of their negotiations. I just can't get invested in this nonsense.
- bigdubs 12y ago
  My read from the post was different; the peering agreements are in place, but one side (the isp's) aren't matching the investment level3 is making into it's infrastructure to handle all that streaming. They are instead asking for $$ in addition to the $$ they collect from their customers, essentially double dipping.
- dpcx 12y ago
  Unless I'm mistaken, by paying for an internet connection, the "fat-cat vc funded video content providers" are paying for "their mp4-based saturation".
- burke 12y ago
  I find it a bit offensive that this can be accurately viewed as a negotiation. This is like demanding the supermarket buy your grocery delivery business a larger van when the toilet paper manufacturer starts selling larger packs. It's arrogant, and violates the proper division of responsibilities. It only might work because of local monopolies. Netflix is already paying for traffic to Level3. ISPs need to upgrade their infrastructure and increase prices if it's too expensive.
  - diminoten 12y ago
    L3 isn't the only content provider, and the 5 UNNAMED ISPs (Comcast isn't necessarily the culprit here) can still provide content to their customers. Netflix is free to hire one of the many CDNs which do play ball with these ISPs.
- knome 12y ago
guelo 12y ago
If the peering ports are congested that means that either the ISP needs to add more ports, or they are oversubscribing their capacity. Just make it illegal to sell more capacity than you have and the problem is solved.
- runeks 12y ago
  > Just make it illegal to sell more capacity than you have and the problem is solved. No. The result would be that you would pay 100 times more for your Internet connection. Or, alternatively, that you cannot get a guaranteed bandwidth, but you pay per GB. This article: http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte... states that Level3 has 13,600 Gbps of capacity with its peers. Your solution would mean that Level3's peers can only sell 13.6 million 1 Mbps connections without overselling their bandwidth. Even if Level3's peers have several other Internet wholesale agreements -- let's say they actually have 5 times that capacity (most likely an overstatement) -- all Level3's peers would only be able to sell 68 million 1 Mbps connections. That would be some expensive 1 Mbps connections!
ChuckMcM 12y ago
There is an interesting unbalance because Comcast has so much leverage by owning the last mile, they can push around Tier 1 providers. I'd like to fix that, mostly by creating a public policy around municipally owned Layer 1 infrastructure between customers in their cities and a city exchange building. Conceptually it would be no different than the city owning the sewers and outsourcing the water treatment plant to a contractor (or two). Creating a new "ISP" would involve installing equipment in the City Exchange(s), providing compatible customer premises equipment to subscribers, and then patching their 'port' at the City Exchange to the ISP's gear. Its going to be a long conversation :-)
- seabee 12y ago
  Sounds like the situation in the UK which largely works quite well.
- exelius 12y ago
  This is how it should probably be done if we were starting from scratch. A system like the one you describe would cost somewhere in the neighborhood of $300-500 billion; which isn't completely insane as far as national infrastructure projects go. The problem is that we're not starting from scratch. It's a lot harder for the government to justify that expense when there are a number of viable privately-owned alternatives. Or to put it another way: if you had $500 billion to spend and you had to choose between overhauling the nation's education system (which desperately needs it) and providing better Internet access to people, which would you choose? Furthermore, do you trust your city to properly maintain said infrastructure? Because I sure as hell don't. All I have to do is look at the pothole-strewn roads outside my house to know that. Fiber optics are awesome now; but what happens in 10 years when we need a new type of quantum fiber made from carbon nanotubes to go faster? Will the cities be able to make the investments? I have my doubts given the overall poor state of municipal infrastructure around the country for things like water and sewage. The system we have now is far from perfect, but we're going to need to figure out how to make something work within the confines of what we have today: a privately owned and operated physical infrastructure controlled by a small number of companies. How do we craft regulation such that those companies are more profitable when they do what we want them to do?
swillis16 12y ago
It will be interesting to see how gaming download services such as Xbox Live, PSN, Steam, etc would be affected as there as the file size of video games gets larger due to advances in the video game industry.
ry0ohki 12y ago
"and make their profit on the Internet because it costs so little to provide once the basic cable plant is built." That's some big hand waving, because laying the cable costs a fortune, and takes many years to recoup the cost which is why there is so few are competing for this "super profitable" business.
- danielweber 12y ago
  It's not even a one-time cost. Without knowing the specific numbers, I'm pretty comfortable saying that the costs to an ISP to get the bits all the way until they reach the peering point is much less than the costs to an ISP to continually upgrade their network.
Havoc 12y ago
> It’s about money and American business, because this is a peculiarly American problem. Hardly. We've experienced the whole interconnect brinkmanship locally too (South Africa). Its actually quite the opposite - the interconnect things are a lot nastier in other countries because it tends to be paid for (powerful co vs underdog) whilst the bigger US setups seem to run mostly open peering.
mncolinlee 12y ago
I can't help but wonder if the RICO Act applies to this sort of extortion. My first thought was FCPA, but none of the ISPs involved can likely be construed as "foreign officials." The behavior can be described as demanding kickbacks, however.
neil_s 12y ago
Since everyone is pitching their own solutions, how about I post mine. Let's take the example of Netflix and Comcast. Instead of no deal with Comcast, and thus giving Comcast Netflix users really slow or no service, Netflix should make the deal for now, and tell subscribers that if you use Comcast the Netflix rental is higher. By passing off the higher costs to the users, Comcast customers are given the incentive to switch ISPs. Everyone shows loss aversion, and so will be determined to find out why being on Comcast gets them penalised. They will learn about its dick moves, and complain to Comcast to make them remove these fees so they can access Netflix, which they have already paid for access to.
- Jtsummers 12y ago
  Netflix would lose, they have competitors and Comcast (and potentially, in the future, ComcastTWC) has too many customers. Increasing the costs to all of them would result in the loss of too many customers before the dust settled.
snambi 12y ago
why there are no last mile providers like comcast and ATT?
- jagger27 12y ago
  Local governments grant monopolies to Comcast or AT&T.
  - Jtsummers 12y ago
    And some state governments force it on the local governments (see NC and TWC versus Wilson, NC).
timr 12y ago
People keep claiming that there should be a "free market" for bandwidth...but then they say that the ISPs should have to absorb the costs of peering (which can be significant -- the hardware isn't free) without passing the costs on to anyone. The backbone providers complain when the cost is passed to them; the consumers scream bloody murder when the costs are passed to them in the form of a bill. Obviously, there's no free market in the status quo: we (consumers) basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure. There's an economic disconnect, and legislating that it shouldn't exist seems worse than futile. I say: pass the costs on to the consumer, and break down the monopolies on last-mile cable service. If the cable companies had to compete for subscribers, they could still pass on the costs of improving their infrastructure, but they'd have to compete with everyone else to do it. In other words, the problem here isn't "net neutrality" -- it's that we've got a monopoly at the last mile that we need to destroy.
- Moral_ 12y ago
  >which can be significant -- the hardware isn't free) without passing the costs on to anyone. I'm not trying to be rube but have you seen how much comcast/verizion/TWC customers pay? It's more than enough to string another switch and fiber line at a route point.
- gph 12y ago
  >basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure Are we though? Our bills have only been increasing. The amount of money they put into infrastructure has decreased[1]. The Telecoms did have to put a lot of initial investment into infrastructure, but that was largely from laying the wires. Most of them are already starting to see positive returns on those investments [2]. They already have the infrastructure to support traffic to consumers in the last mile. The peering infrastructure being upgraded shouldn't cost them even 1/10th as much. So why should they get to pass their peering infrastructure costs on to the content providers? They should pass it on to the consumer if they must, after all it's us trying to get the content. But really they are going to be making such a large profit off us over the coming decades it's ridiculous for them to cry poor. Net neutrality is fairly meaningless when it comes to the last mile. I doubt hardly anyone is saturating their last mile worth of bandwidth. I regularly have three streams going in my household and it doesn't saturate my bandwidth. That part isn't what needs upgrading. [1]http://www.vox.com/2014/5/12/5711082/big-cable-says-broadband-investment-is-flourishing-but-their-own-data http://www.vox.com/2014/5/12/5711082/big-cable-says-broadban... [2]http://dailycaller.com/2013/02/15/does-cable-really-have-a-97-profit-margin/ http://dailycaller.com/2013/02/15/does-cable-really-have-a-9...
fragsworth 12y ago
The proposed solution is at the bottom of the article (which is why everyone seems to have ignored it): > The solution to this problem is simple: peering at the original NSFnet exchange points should be forever free and if one participant starts to consistently clip data and doesn’t do anything about it, they should be thrown out of the exchange point. I do have a couple questions though - who is in charge of the original NSFnet exchange points, and do they have this authority?
jamesbrownuhh 12y ago
The UK experience is, broadly speaking, that any ISP who is sufficiently tall to have the appropriate interconnects can offer a service to a customer via the incumbent's last-mile infrastructure. (This is for telephone-based ADSL broadband - the UK's only cable operator is not bound by this.) But, furthermore, competitor ISPs are enabled to install their own equipment and backhaul directly into local exchanges, known as LLU - local loop unbundling. LLU allows competitor companies to provide just your broadband, or your voice telephone service, or both. There is one further step, whereby the prices of the incumbent monopoly are regulated in areas where no competition exists. Ironically this works in the opposite way to how you'd think, as it forces the incumbent NOT to offer their lowest prices in that market - the intention being to make monopoly areas prime targets for competition and to ensure that potential competitors aren't scared out of the area by predatory pricing. It's an odd system with good and bad on both sides, but it seems a lot better than being stuck with a single source of Internet access.
jrochkind1 12y ago
You would think, okay, if Comcast is terrible at maintaining sufficient peering for it's customers needs -- and if the OP proposal to throw Comcast out of peering exchange points happened, that would certainly lead to increased terribleness for it's customers -- then eventually it's customers would choose a different ISP. The market would solve it. The problem is that in many many markets, Comcast (or another ISP) are pretty much the only choice. Customers don't have another option, no matter how much Comcast underfunds it's peering infrastructure or gets thrown out of peering exchange points. So what is the consequence to Comcat for underfunding? What is the consequence to Comcast for even such a disastrous outcome as getting kicked out of the peering exchange point? Not a lot. I'm not sure what the solution is, but 'regulate them as a common carrier' is certainly part of it, since they are a monopoly, and the common carrier regulatory regime was invented for exactly such a monopoly.
- sadris 12y ago
  > Comcast (or another ISP) are pretty much the only choice. Customers don't have another option If an entire town is angry, their city councilors will very quickly utilize eminent domain on the fiber in the ground and sell it to a competitor.
  - kej 12y ago
    But not before the ISP lobbies the state legislature to decree that cities can't annex fiber.
guardiangod 12y ago
I don't know why everyone is up in arm over this. Here is reverse thinking and a perfect oppoturnity for everyone. The current situation is that Comcast doesn't have the equipment/resources to handle extra internet traffic at its peers. Most people want Comcast to buy more stuff to handle it, why don't we think the opposite way- get Comcast to decrease its amount of traffic? If we can get Comcast to consume less traffic, they wouldn't have to complain to other peers about load asymmetry. The best way to decrease traffic? Make Comcast has less customers. Why does Comcast has so many customers, even though their resources cannot handle it? Because they have a government mandated monopoly in the last mile, so they are forced to have more customers than what they can handle. We can come to a conclusion that last-mile monopoly -> network congestion -> forcing L3 to pay for peer. If Comcast has to compete with other ISPs for last miles, the traffic load would shift from 1 single entity (Comcast) to 10+ smaller ISPs. In such case the traffic load problem would not exist. Another solution is to breakup Comcast. See? This is a perfect opportunity. Comcast can has its multi-tier network, but at the price of the last mile monoploy. After all, if they want to have the right to choose peers, we customers should also have the right to choose ISPs.
cobookman 12y ago
I've previously interned at one of the mentioned Cable Companies, and I see both sides. The solution is to make it 'capitalistic'. Change all of our internet contracts from Unlimited (up to 'x' GB/month), to a simple $/gb cost. It would be in the ISPs best interest to provide their customers the fastest internet connection as possible. E.g, if a customer can stream a 4k video vs SD then the ISP would make more money per unit time. Think of it this way, if comcast charges $0.25/GB, and a netflix SD show is say 1GB and HD is 4GB, then comcast grosses $1 for HD and $.25 for SD for the same customer streaming request. Over time its likely the price per GB would decrease, just like it has for cellular. On a more evil side, this would also stop chord cutters. Pirating content is no longer 'free', and Netflix would cost significantly more than $10/month ($10/month + 'x'GB * $/GB). As for what rates to expect, if comcast charges in ATL $30-55 for 300GB, that'd be about $.10/GB to $.20/GB. As for speed tiers in a $/gb system, your guess is as good as mine.
- seehafer 12y ago
  How does changing plans from unlimited to metered data make them more 'capitalistic'? An example of making things more capitalistic would be to make illegal the municipal contracts granting monopolies on last mile service.
  - cobookman 12y ago
    In an unlimited system, a customer which uses 10GB/month grosses the company the same amount as a customer who uses 1TB/month. Going to a metered system, people who use more bandwidth are charged more. Technically ISPs don't pay for bandwidth, but for a 'x' GBps link. But our electric companies do the same, they pay for 'x' Kw/h (a power plant can only output so much electricity per hour). >...make illegal the municipal contracts granting monopolies on last mile service. Thats a very hard peice of legislation to pass. I'm not even sure if Google Fiber would agree to this. Even if this legislation passed, would that fix our net neutrality rules? Moving to a metered system gets rid of most of the ISPs arguments against net neutrality....all of them except having a company pay for your bandwidth.
    - 12y ago
rossjudson 12y ago
Level3 should drop the same percentage of outbound packets from Comcast, that Comcast drops on the inbound. If every tier 1 did, Comcast's internet service wouldn't look all that good any more, would it?
jvdh 12y ago
Just for scale, backbone links these days are not 10 gigabits/sec, more in the order of 40-100 gigabits/sec. The Amsterdam Internet Exchange is the largest and most important exchange in Europe, and it's peak traffic each day reaches 3 terabits/sec.
mokus 12y ago
> Nobody paid anybody for the service because it was assumed to be symmetrical: as many bits were going in one direction as in the other so any transaction fees would be a wash. The justification for peering is not equal traffic, it's equal value - my customer wants to communicate with your customer. Regardless of the direction of traffic, the traffic is equally valuable to both of us because the traffic is the primary thing our customers are paying us for. Unless, of course, I can get you to pay me for it anyway because of some unrelated advantage such as the fact that your customers can leave you more easily than mine can leave me. Comcast and others are attempting to leverage exactly that - in many regions they have no viable competition whereas Netflix and L3 are much more replaceable in their respective markets. This is a prime example of abuse of a monopoly.
Rezo 12y ago
The extreme download vs upload traffic asymmetry between Comcast and L3/Netflix has been mentioned several times as a straw man argument for why Comcast is justified in charging Netflix directly. Maybe Netflix could find some creative uses for all that idle viewer upload capacity to reduce the deficit ;) - Have every Netflix client cache and serve chunks of the most popular streams P2P-style. You could have a DHT algorithm for discovering chunks or have Netflix's own servers orchestrate peer discovery in a clever way, for example by only connecting Comcast customers to peers physically outside of Comcast's own network. This would reduce Netflix's downstream traffic and increase viewer uploads. - Introduce the Netflix-Feline-Image-KeepAlive-Protocol, whereby every Netflix client on detecting a Comcast network uploads a 5MB PNG of a cat to Netflix's servers over and over again while you're watching a video. Strictly for connection quality control purposes of course.
tom_jones 12y ago
Along these lines, can someone ask whether net neutrality ever existed at all? Akamai and F5 have been helping big corporations like Disney circumvent internet bottlenecks for over a decade now. Those who have had the money have managed to purchase faster delivery schemes for over a decade. Could it be, then, that telecommunications companies are consolidating so that they can extort money not from the small guys, but from the big guys? Are Hulu, Netflix and others willingly submitting to the extortion because they see no other way out? To be sure, the telecommunications industry is in desperate need of regulation because providing good service at a reasonable price for a reasonable profit is not good enough for them.
eb0la 12y ago
The problem is not to peer or not to peer. The problem is WHERE to peer. I work for a european ISP and the problem we have is the location of the peering. Big content providers will happilly peer with you in, say, Palo Alto or Miami; but they will refuse to add a peering connection in Europe. Why? because today the problem is about WHO pays the Intercontinental route (which limited and is expensive bandwidth). Level3 is known in the industry as a pioneer for bit-mile-peering agreements. This means you have to sample the origin and destination of the IP packets and make some calculations to know how many miles the packet has traveled and pay / get paid if someone dumps long haul traffic to a peer. Getting to this is complicated with current tecnhology and many companies are refusing to peer with Level3 because they don't know what will happen with their business with bit-mile-peering agreements.
Eye_of_Mordor 12y ago
Lack of competition all around - just break up the big boys and everything will be fine...
GregFoley 12y ago
The problem would disappear if ISPs used metered pricing. Why do we have unlimited commercial broadband?

781 points by mortimerwax 12y ago

This wouldn't happen if those ISPs didn't have local monopolies. Networks should be opened by selling traffic wholesale to other companies so that they can compete for subscribers on those networks. The network owners would have more than enough money for upgrades and if they don't, downlevel ISPs will sue them.

kasey_junk 12y ago
I'm not making a claim about the veracity of the counter argument to this, but it is easy to state. If network providers had to sell traffic wholesale to other companies, they would have less incentive to upgrade them as those networks then become pure commodities. Right now the local monopolies can use their networks to sell high margin services (cable) bundled with low margin services (internet traffic). If they had to compete with other companies, namely bargain ISP offerings, the cost of ownership of the network would be non-profitable. I've heard some argue that this is precisely why DSL has lagged so far behind cable in upgrades.
- rlpb 12y ago
  > they would have less incentive to upgrade them as those networks then become pure commodities Pit DSL against cable, as has happened in the UK. The POTS network is owned by one company (BT) and the cable network by others. Retail customers generally have a choice of connecting to the Internet by either. BT (POTS network owner) are required to sell traffic wholesale to competitor ISPs (who then buy their own transit). This seems to work very well here, and DSL upgrades continue.
  - kasey_junk 12y ago
    That describes almost precisely how the US system works.
    - opendais 12y ago
      [text removed since it seem to have confused the person who replied] http://news.cnet.com/FCC-changes-DSL-classification/2100-1034_3-5820713.html http://news.cnet.com/FCC-changes-DSL-classification/2100-103... "The ruling puts phone companies on the same regulatory footing as cable companies, which are exempt from having to offer access on their infrastructure to competing Internet service providers."
      - kasey_junk 12y ago
        I'm not an expert, but I'm reasonably certain that Cable providers in the UK are also not required to provide ISP access via their networks.
        rlpb 12y ago
        No, they are not. Otherwise I'd be using one.
        opendais 12y ago
        Yes but he is claiming the US system works like the British system...when it doesn't.
  - robk 12y ago
    British DSL is far worse than cable in my experience. Virgin's cable & fiber offering is massively faster and has a wider footprint than DSL products I've seen.
    - rlpb 12y ago
      > British DSL is far worse than cable in my experience. Define "worse". What ISP were you using? From a peering point of view as described in the article, there are a large number of ISPs buying transit and then selling Internet connections through BT-provided last mile DSL lines. I never suffer from peering congestion, since my (DSL) ISP pride themselves on not having any. For the link from my local exchange to my ISP, there are multiple options, too, and my ISP monitors congestion on my line closely. I currently have no congestion there, either, and if I did, there are multiple providers available (thanks to LLU): http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-purpose.html http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-purpose.ht... The only issue with DSL in the UK is the (mostly analogue) quality of the DSL line itself, and the way that BT (the company with the monopoly on POTS lines) manages them. And perhaps the pace of upgrades (eg. to fibre to a street cabinet and copper from there, instead of copper all the way from the exchange), but upgrade rollouts are happening (and fibre to the cabinet is already available to me).
  - 7952 12y ago
    TV is also a huge factor in the UK. Both BT and Sky need high broadband speeds to be able to offer video on demand that can compete with virgin.
- jbooth 12y ago
  The article said that their cable business is way less profitable than their internet business. I haven't done any research on the topic, but given that they have to pay content providers for cable, and don't for internet, I'm inclined to believe Cringely -- even though in a normal market it should be a low-margin business, selling undifferentiated bits for $.
  - danielweber 12y ago
    If it was "selling bits for $" we would already be done. But today's ISP capacity is unsuitable for tomorrow's ISP demands. One might hand-wave the network as "sunk costs" but the ISP has to keep on sinking those costs. In fact, that's what Crinkley is complaining about: the ISP isn't sinking costs into its peer with L3.
    - jbooth 12y ago
      Sure, so you've got depreciating capital costs on a schedule coupled with ongoing operational costs, that gets you to some amount of $/yr to operate the network, and $/year/bit in terms of network capacity. What should be happening is competition driving down the price per bit to some amount just marginally above their costs. These costs may be high but the margins should be low. That's Econ 101 and it's what everyone who ever advocated for the free market is making a case for. What appears to be happening is they're collecting a big fat surplus on top of that figure, and that surplus is in addition to all the wasted money in their corporations that's never had to be cut in a down quarter.
- opendais 12y ago
  And those people are completely 100% wrong. DSL got screwed for around 10 years by having more expensive regulatory costs than Cable. Why yes, if you tax X more than Y, Y is going to have the advantage.
yk 12y ago
We have essentially such a system in Germany, does not work well. Thing is, that the owner of the network has then to compete with the resellers, but the resellers need to play nice with the owner. The result is a total mess of regulation to force the network owner ( T-Systems) to play nice.
- a3n 12y ago
  Back in the dial-up days I had my choice of ISPs. Some people chose on price, I chose on quality/service, and mine wasn't that much more than the cheapos. I can't recall any issues with the underlying phone company operator of the wires, although that may be because the wires were primarily for phone and they have to work by law (I think). I miss the days of being able to choose my ISP. Now I have Comcast, with CenturyLink DSL an inferior possible second choice.
- NorthGuy1 12y ago
  We have the same system in Canada and the wholesaler ISP (Bell) famously imposed unpopular usage based billing on the re-seller (TekSavvy) to kill the only advantage it had over the ISP. http://www.cbc.ca/news/technology/crtc-approves-usage-based-internet-billing-1.879213 http://www.cbc.ca/news/technology/crtc-approves-usage-based-...

JoshTriplett 12y ago

Somebody has to pay the money to upgrade the equipment and bandwidth available at these exchange points. The very reasonable argument in this article is that the ISPs should pay that cost, which seems reasonable given that their customers are demanding it. It sounds like the ISPs are playing a game of chicken, trying to see if their peers like Level3 will throw money in to pay for the ISP to upgrade its equipment and bandwidth. That's certainly something the ISPs can try to do; on the other hand, what are their customers going to do, not use Netflix and YouTube? If a pile of customers of one ISP start reporting that they're all having a poor experience with high-bandwidth video, and there are a pile of well-publicized press releases blaming the ISP, customers will start complaining to the ISP, and they'll have to upgrade their infrastructure eventually. (And in areas where they have competition, there's an incentive to upgrade before the competitors, to avoid losing customers; while there isn't such competition in every locale, there are enough locales with more than one ISP choice to make those customers painful to lose.) But what does any of that have to do with mandated peering requirements at the NSFnet exchanges? Who would enforce that, and why, when any two major networks can set up peering at any number of meet-me rooms? Requiring that an ISP peer as much traffic as is available or not peer any at all seems ridiculous; some ISPs will suck more than others, but that's the problem of them and their customers, not a problem for the entire Internet. Meanwhile, I'm surprised there aren't more startups and VCs looking to bet that "new ISP that doesn't suck" is a viable business model. People are chomping at the bit for Google Fiber, which seems unlikely to grow to a national level without developing competitors. This is a space with very few competitors, and there hasn't been serious competition in that space since DSL stopped being a viable option.

bsimpson 12y ago
Google's ability to print money in conjunction with its generally high PR ratings give it the cache to take on a project like fiber. I can't imagine it's cheap or easy to lay fiber if you don't have substantial financial backing as well as a political goal you're trying to achieve. That said, I've used both Sonic.net and MonkeyBrains here in San Francisco. Never had to rely on a major ISP in this town, and I like it that way.
- 12y ago

xhrpost 12y ago

I like the article overall but I don't understand the author's proposed solution. The issue as it stands is apparently a lack of peering, in that big ISPs are using transit to reach large content providers rather than directly peering to those networks. So how would "kicking them out" for a maxed out connection work? If I buy transit from Level3 and my connection maxes out, I'm no longer allowed to be a customer of Level3?

logfromblammo 12y ago
I think the proposal is to simply sever the link when the peering connection saturates. The traffic out through the link from LazyISP is generating more traffic in through the link than the pipe can handle. LazyISP pretends that this is not their problem, that the Internet is somehow trying to shove its traffic through poor ol' LazyISP's pipes. The claim is analogous to traffic exiting a Autobahn-style limited-access highway and attempting to reach its destination faster by taking the narrow country backroads to get to the on-ramp of the next highway. The only real reason for any traffic to route to LazyISP is if one of the endpoints is in there, and there is seldom a reason for that traffic to go anywhere but the shortest route between the end point and the nearest backbone highway access point. The typical argument from LazyISP is so ridiculous on its face that it makes me furious every time I see it. All of those guys present asymmetric upload and download speed to residential customers. Every last one of them. And they pretend that when all of their customers' asymmetric bandwidths are summed up, the total is somehow magically supposed to be symmetric. Every business on the planet whose cash flow depends on pushing bits out to the Internet goes directly to the backbone with content delivery networks, or at least to an ISP that is serious about peering. They are not going to an ISP with crappy interconnects. There is no way in hell that any cul-de-sac on the network map like a last-mile ISP is ever going to push as many bits to the backbone network as it pulls from it, especially when they do nasty tricks like throttling, port blocking, and hidden metering. The point is that a steaming pile like Time Warner does not have any internal content providers that any of its customers want to connect to, precisely because of its business practices. If Time Warner cannot connect to the Internet, it has nothing to offer the customer through that precious last mile connection except television channels. So they stop paying the $25-150 a month and try to make do with standing on a ridge waving semaphore flags at each other. LazyISP needs Level 3 more than Level 3 needs them. Therefore, when LazyISP tries to extort them, the correct response should be, "Fuck you," as they flip the off switch.

rrggrr 12y ago

Godaddy, Rackspace, Google, Amazon, etc. have skin in this game. With multiple redundant network connections they could, for a day or a week, defend neutrality by shaping their traffic to the lowest common denominator or routing their traffic to avoid the peer's punitive bottlenecks. Today its Level 3 and Netflix, but tomorrow it could easily be them.

signet 12y ago

If a customer is paying for an internet connection, they are paying for access the full internet, to the best of their ISP's abilities. This is the net neutrality law we need: ISPs should be compelled to upgrade their backbone links as they become congested, to satisfy their customer's demand. Congestion can be easily monitored and often these peerings are "free". (Yes there is a non-zero cost to increase switch and router capacity and to have someone plug the cables in, but it's not like Level3 is charging for the bits exchanged.) But the point is, since most ISPs are de-facto monopolies in this country, we need rules telling them they have to upgrade their capacity to meet their customers demand, if they are promising broadband speeds.

siculars 12y ago
This. ISP's of the like mentioned by the OP are being paid for INTERNET access. Not access to the local network resources on the ISP's network. If I pay for an X up/Y down connection to the internet then that is what I expect. I expect X up/Y down to any host/content on the internet (given that host can reply at that speed, etc.)
mattmcknight 12y ago
We probably need pricing more directly based on bandwidth consumed, maybe with peak hour pricing, as opposed to just based on "speed" with high bandwidth caps.
- morgante 12y ago
  Don't buy into the cable PR. They have incredibly high margins and could easily make upgrades without further gouging customers.
  - rgbrenner 12y ago
    "incredibly high"? Comcast and Time Warner have profit margins of 8-10%. For comparison, Target has a profit margin of 3%, and Google's is 21%. Sounds more mid-range to me. As for your second point, that they could easily make upgrade without additional charges.. have any numbers backing that up? The last mile is expensive.

deleted 12y ago

[deleted]

spindritf 12y ago

Except it’s actually right (not wrong) because those bits are only coming because customers of the ISPs — you and me, the folks who have already paid for every one of those bits — are the ones who want them. What is the source of the notion that, because you paid for your consumer broadband, all bits are paid for and the charge for carrying them cannot be split with the other side of the connection? Why is it so bizarre that both sides of the connection have to pay for it? Because you're used to your phone working differently? As an analogy, you know how you used to pay for a subscription to a magazine and there were ads inside which advertisers (the other side of the connection via the magazine in this case) also paid for? The magazine split its fee in two: you paid part of it, and the advertisers paid the other part. It's the same here. There is nothing fundamentally wrong with charging both sides. You may prefer a different fee structure but a better argument than "I already paid for it!" is necessary.

wging 12y ago
In this case, the difference is that broadband subscribers are not being sold a magazine which they know will contain ads--they are being sold a full connection to the internet! That's the precedent, that's what we think we're getting for our money.
- spindritf 12y ago
  What does full connection to the Internet mean? No ISP can guarantee it because any other independent network can drop traffic at will. And the Internet is just a bunch of independent networks. The service you believe you paid for doesn't and cannot exist. I can filter out all Comcast IPs on my firewall tomorrow if I feel like it. Many people actually do it to third world providers.
  - talmand 12y ago
    There's a difference between bits not being delivered because of the originating server failing to deliver and my ISP actively preventing delivery. "Unlimited Internet" has already been established in sales pitches for years, the language defined once people understood the concept of bandwidth caps. "Unlimited" means you have unlimited "access", not unlimited "bandwidth". This has been the industry's very own sales pitch for years.

api 12y ago

While I agree with the general thrust of the article, there is one fallacious argument here. Cringely argues that cable breaks even and money is made on the net, but that's an artificial distinction. What if cable disappeared? Would they still make money if they had to pay for the upkeep of the network with only Internet fees? The desperation and risk of this game of chicken convinces me that the answer might be "not much." The loss of cable might very well be apocalyptic for these companies, at least from a shareholder value and quarterly growth point of view. What's happening is very clear to me: the ISPs are trying to either harm the Internet to defend cable or collect tolls on streaming to attempt to replace cable revenue. That's because cable is dying a slow death. This is all about saving cable. The fundamental problem is that cable ISPs have an economic conflict of interest. They are horse equipment vendors that got into the gas station business, but now the car is driving out the horse and their bread and butter is at stake.

Jtsummers 12y ago
> Would they still make money if they had to pay for the upkeep of the network with only Internet fees? In parts of the country, slower-speed copper, fast-download cable, and a few fiber networks are already built out. The cable distribution giants like Time Warner Cable and Comcast are already making a 97 percent margin on their “almost comically profitable” Internet services, according to Craig Moffet, an analyst at the Wall Street firm Bernstein Research. As Levin points out, “If you are making that kind of margin, it’s hard to improve it.” And most Americans have no choice but to deal with their local cable company. -- http://www.technologyreview.com/news/510176/when-will-the-rest-of-us-get-google-fiber/ http://www.technologyreview.com/news/510176/when-will-the-re... If that's accurate, I suspect that they could live without cable as we know it today and still make plenty of money. Working on finding out more numbers to see what the revenue breakdown is between cable and internet, though I wouldn't mind if someone else chimed in. EDIT: Found this [1] for Comcast. For 2012 and 2013 they've lost video customers, increased internet customers. Had growth in revenue for both categories, but a greater increase from internet (percentage and absolute). So the way things are heading, cable leaving would cost them a lot. However, they can probably make up for it elsewhere, and, ultimately, should be planning on cable TV going the way of the dodo regardless of the nearterm revenues and profits. It's not a viable longterm strategy, like SMS (while immensely profitable) for cell carriers, a good short term strategy but as content shifts to the generic data channels the specialized channels will fall into disuse. [1] http://files.shareholder.com/downloads/CMCSA/3182774232x0xS1193125-14-47522/1166691/filing.pdf http://files.shareholder.com/downloads/CMCSA/3182774232x0xS1... EDIT 2: Check out page 53 of the linked filing. Video revenue for 2013 was $20.5 billion. Programming costs were $9.1 billion. So the net, ignoring any other costs, right there is $10.4 billion. If the ISP side of the house is actually making 97% margins like the claim in the Technology Review article, then it is making almost as much as the video side of the house. At the current growth rates it'll surpass the video profit, but not revenue, sometime in the next few years. Their NBCUniversal acquisition may reduce their programming costs, I'm not sure of the legalities involved in that (their growth and acquisitions could put them afoul of anti-trust/monopoly laws and regulations).

sbierwagen 12y ago

Regulate ISPs as utilities.

hga 12y ago
Common carriers are the words of art for these sorts of utilities. And Comcast and company are in steadily increasing danger of this as they piss off more and more voters (at a certain point, all the money in the world is of no interest to a politician who perceives himself in danger of having to spend more time with his family after the next election; gun owners have demonstrated this many times).

lifeisstillgood 12y ago

1. Peering is based on equal traffic both ways. At the moment we tend to download gigabytes with a few bytes of request. As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise 2. rise of ad-hoc local networks This might come out of mobiles, this might be me dreaming, and it might come with sensible home router designs, but ultimately most of the traffic I care about probably originates within 2 miles of my house - my kids school, traffic news, friends etc A local network based on video comms - that will never happen. just like mobile phones. 3. electricity and investment In the end this is down to government investment. Let's not kid ourselves, gas, water, electricity, railroads, once they passed some threshold of nice to have into competitive disadvantage not to have, governments step in with either the cash or the big sticks. Fibre to the home, massive investment in software engineering as a form of literacy, these are the keys to the infrastructure of the 21C and it's a must have for the big economies, and it's a force multiplier for the small.

dragonwriter 12y ago
> As video-communications really takes off (yes chicken and egg - see below) this will get lost in the noise Aside from the chicken and egg problem, I don't think you ever get to a point where that really is the case. Not because video comms never take off, but because even when it does its a small share of content delivery and the net is still wildly asymmetrical.
ep103 12y ago
The problem is your point #3 has never been true, though. In each case, in American history (though I don't know about water), the utility was privatized for _years_, became local monopolies, and used that monopolistic advantage to become one of the most major Washington lobbying industries of the time, often to the massive detriment to specific sections of the economy. In each case it was only _after_ an alternative industry rose in lobbying prominence in washington (often, the next one on your list) that the previous utility lost their position as #1 lobbying firm in Washington, that their competitors were able to then force anti-trust lawsuits, and or get fair government regulation. The railroads were monopolized until the late 1800s, the original American idea of a fat cat comes from railroad owners who made profits at the expense of midwestern farmers. That was eventually regulated when oil became a major lobbying industry, whereupon oil and gas subsidies became the new norm and we regulated the railroads. Electricity is a general complex area, but outside of domestic coal-based electricity production, my understanding is that the majority(?) of American electricity comes from nuclear power, which in turn is a regulated monopoly. That industry is so locked that Northern America imports much of its electricity from nuclear plants in Canada, and Texas is actually on Mexico's electricity grid. I was somewhat hopeful that as Wallstreet rose to replace other industries as one of the top lobbying firms in America that they would end up pushing for net neutrality so as to commoditize their network costs, but instead they simply pushed for B2B fiber to be regulated completely differently (and more sanely) than consumer networks. I don't want to wait for something better than the internet to come around, before we get net neutrality back.

mhandley 12y ago

Would be very interesting to see what happened if the big CDN providers just depeered Comcast for 24 hours. Would certainly cost Comcast a fair amount in customer support calls, bad publicity, and properly bring the debate to the general public.

klodolph 12y ago
It would also give the CDNs a fair amount in customer support calls, bad publicity, and make their paying customers angry about lost business.
- ep103 12y ago
  I don't know about bad publicity though. To each customer support call simply say, Comcast has broken their contract with our CDN. Please call comcast. Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin, people will easily believe that, especially since in this case, it would be true. Level3 et all are enabling this behavior.
  - chimeracoder 12y ago
    > Considering Comcast is one of the two most hated companies in America right now, by a _huge_ margin And Time Warner is the second: http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-survey-study/ http://bgr.com/2014/05/20/comcast-twc-customer-satisfaction-... Nobody likes the cable companies. It would be very easy for someone to capitalize on that hatred to win public support for net neutrality; it's just a matter of crafting the right campaign.
    - coldpie 12y ago
      This is why I'm so disappointed to see the small-medium ISPs hop right in bed with Comcast and co-sign anti-net neutrality letters: http://arstechnica.com/business/2014/05/top-isps-threaten-to-be-less-innovative-and-spend-less-on-network-upgrades/ http://arstechnica.com/business/2014/05/top-isps-threaten-to... With the national attention this issue is getting, this is a tremendous marketing opportunity to take customers away from the big dogs.

keehun 12y ago

Maybe I'm naïve beyond any recognition, but shouldn't the ISP's or whoever is peering charge based on the bandwidth amounts? It sounds like they have a flat-rate contract with each other and now they're charging more?

awor 12y ago
From the article, it seems like the "flat-rate contract" the ISPs have for peering with L3 is $0. The issue that L3 is raising, is that the equipment/appliances that the ISP has installed in these peering arrangements are insufficient to keep up with the amount of traffic which the ISP's customers are requesting, and L3 is attempting to provide. If the ISPs were to upgrade their peering equipment to handle their customer's requests, there would be no issue.
- keehun 12y ago
  So by Netflix making a deal with Comcast, did L3 effectively get cut out of the picture? Also, isn't increasing equipment/bandwidth part of the business? It seems so ridiculously absurd that Comcast is balking to increasing its bandwidth...
hga 12y ago
Inexperienced in the field? This sort of thing has been going on since at least the '90s; this search ought to bring up some relevant hits: https://www.google.com/search?q=uunet+peering+ma-east https://www.google.com/search?q=uunet+peering+ma-east One thing to focus on is that there are 2 customers, and two middlemen. E.g. Netflix <-> Level 3 <-> Comcast <-> a Netflix subscriber. Netflix and the Netflix subscriber are each paying their ISPs hefty sums for access to "the Internet". In the ideal world, Level 3 and Comcast would peer at various points for maximum efficiency on both their networks. Comcast enjoys a local municipality government monopoly in their markets, with iffy competition from state and Federal monopolies granted to the RBOCs Verizon, AT&T, and in theory CenturyLink (which owns the US West states, but has never been very serious about Internet service and especially their own "TV"/cable service), and all the small ILECs who provide telephone service where the big boys had no interest. One big problem here is that "TV"/cable service deployment for the first two RBOCs, who cover the vast majority of the population, has been halted outside of contractual requirements with municipalities and the like. (If you're in a really bad spot, like I am, on unincorporated land adjacent to two cities, the local cable company is not required to provide service and doesn't, AT&T provides slow and expensive (but reliable!) ADSL with very small caps, 150 GiB per month ... and now they're buying the DirectTV my parents use to get around that cable company problem!) So as noted in the article, the bandwidth flow might be asymmetrical, and for these local "last mile" ISPs, there are technical and economic reasons for only providing asymmetrical service in the first place, e.g. head end noise, but all the Netflix bits coming in through Level 3 to Comcast Netflix subscribers are paid for, by the hefty payments the subscribers pay to Comcast for access to "the Internet". (And which are soon going to get caps as well from what I've read.) But Comcast would of course prefer you get your "TV"/cable bits from them, and have no particular interest in enabling Netflix to supplant them. And they're now committing fraud on their subscribers, by telling them they're providing access to "the Internet", but deliberately degrading significant portions of it.

pessimizer 12y ago

This cable-menu style image from the comments is scary: http://i.huffpost.com/gen/1567010/original.jpg http://i.huffpost.com/gen/1567010/original.jpg

qnaal 12y ago
Fearmongering- the internet is not going to be that easy to kill.
- pessimizer 12y ago
  If they stop supplying it to us without discrimination it will. The internet is made up of plastic and metal boxes that are under other people's control - not magic.
  - sliverstorm 12y ago
    Things that cannot be simply cut off: - populist revolutions Things that can: - electricity - the internet
wmf 12y ago
But the clever thing about Comcast's behavior is that the customer does not see anything like that image.

droopybuns 12y ago

So on the one side is the fat-cat ISP who doesn't want to make expensive capital investments ih their transport. And on the other side is the fat-cat vc funded video content providers, who don't want to pay for the their mp4-based saturation of all the pipes. This is a negotiation. There are two active media campaigns that are trying to gin up our anger against The Other Guy (tm) as part of their negotiations. I just can't get invested in this nonsense.

bigdubs 12y ago
My read from the post was different; the peering agreements are in place, but one side (the isp's) aren't matching the investment level3 is making into it's infrastructure to handle all that streaming. They are instead asking for $$ in addition to the $$ they collect from their customers, essentially double dipping.
dpcx 12y ago
Unless I'm mistaken, by paying for an internet connection, the "fat-cat vc funded video content providers" are paying for "their mp4-based saturation".
burke 12y ago
I find it a bit offensive that this can be accurately viewed as a negotiation. This is like demanding the supermarket buy your grocery delivery business a larger van when the toilet paper manufacturer starts selling larger packs. It's arrogant, and violates the proper division of responsibilities. It only might work because of local monopolies. Netflix is already paying for traffic to Level3. ISPs need to upgrade their infrastructure and increase prices if it's too expensive.
- diminoten 12y ago
  L3 isn't the only content provider, and the 5 UNNAMED ISPs (Comcast isn't necessarily the culprit here) can still provide content to their customers. Netflix is free to hire one of the many CDNs which do play ball with these ISPs.
knome 12y ago

guelo 12y ago

If the peering ports are congested that means that either the ISP needs to add more ports, or they are oversubscribing their capacity. Just make it illegal to sell more capacity than you have and the problem is solved.

runeks 12y ago
> Just make it illegal to sell more capacity than you have and the problem is solved. No. The result would be that you would pay 100 times more for your Internet connection. Or, alternatively, that you cannot get a guaranteed bandwidth, but you pay per GB. This article: http://blog.level3.com/global-connectivity/observations-internet-middleman/ http://blog.level3.com/global-connectivity/observations-inte... states that Level3 has 13,600 Gbps of capacity with its peers. Your solution would mean that Level3's peers can only sell 13.6 million 1 Mbps connections without overselling their bandwidth. Even if Level3's peers have several other Internet wholesale agreements -- let's say they actually have 5 times that capacity (most likely an overstatement) -- all Level3's peers would only be able to sell 68 million 1 Mbps connections. That would be some expensive 1 Mbps connections!

ChuckMcM 12y ago

There is an interesting unbalance because Comcast has so much leverage by owning the last mile, they can push around Tier 1 providers. I'd like to fix that, mostly by creating a public policy around municipally owned Layer 1 infrastructure between customers in their cities and a city exchange building. Conceptually it would be no different than the city owning the sewers and outsourcing the water treatment plant to a contractor (or two). Creating a new "ISP" would involve installing equipment in the City Exchange(s), providing compatible customer premises equipment to subscribers, and then patching their 'port' at the City Exchange to the ISP's gear. Its going to be a long conversation :-)

seabee 12y ago
Sounds like the situation in the UK which largely works quite well.
exelius 12y ago
This is how it should probably be done if we were starting from scratch. A system like the one you describe would cost somewhere in the neighborhood of $300-500 billion; which isn't completely insane as far as national infrastructure projects go. The problem is that we're not starting from scratch. It's a lot harder for the government to justify that expense when there are a number of viable privately-owned alternatives. Or to put it another way: if you had $500 billion to spend and you had to choose between overhauling the nation's education system (which desperately needs it) and providing better Internet access to people, which would you choose? Furthermore, do you trust your city to properly maintain said infrastructure? Because I sure as hell don't. All I have to do is look at the pothole-strewn roads outside my house to know that. Fiber optics are awesome now; but what happens in 10 years when we need a new type of quantum fiber made from carbon nanotubes to go faster? Will the cities be able to make the investments? I have my doubts given the overall poor state of municipal infrastructure around the country for things like water and sewage. The system we have now is far from perfect, but we're going to need to figure out how to make something work within the confines of what we have today: a privately owned and operated physical infrastructure controlled by a small number of companies. How do we craft regulation such that those companies are more profitable when they do what we want them to do?

swillis16 12y ago

It will be interesting to see how gaming download services such as Xbox Live, PSN, Steam, etc would be affected as there as the file size of video games gets larger due to advances in the video game industry.

ry0ohki 12y ago

"and make their profit on the Internet because it costs so little to provide once the basic cable plant is built." That's some big hand waving, because laying the cable costs a fortune, and takes many years to recoup the cost which is why there is so few are competing for this "super profitable" business.

danielweber 12y ago
It's not even a one-time cost. Without knowing the specific numbers, I'm pretty comfortable saying that the costs to an ISP to get the bits all the way until they reach the peering point is much less than the costs to an ISP to continually upgrade their network.

Havoc 12y ago

> It’s about money and American business, because this is a peculiarly American problem. Hardly. We've experienced the whole interconnect brinkmanship locally too (South Africa). Its actually quite the opposite - the interconnect things are a lot nastier in other countries because it tends to be paid for (powerful co vs underdog) whilst the bigger US setups seem to run mostly open peering.

mncolinlee 12y ago

I can't help but wonder if the RICO Act applies to this sort of extortion. My first thought was FCPA, but none of the ISPs involved can likely be construed as "foreign officials." The behavior can be described as demanding kickbacks, however.

neil_s 12y ago

Since everyone is pitching their own solutions, how about I post mine. Let's take the example of Netflix and Comcast. Instead of no deal with Comcast, and thus giving Comcast Netflix users really slow or no service, Netflix should make the deal for now, and tell subscribers that if you use Comcast the Netflix rental is higher. By passing off the higher costs to the users, Comcast customers are given the incentive to switch ISPs. Everyone shows loss aversion, and so will be determined to find out why being on Comcast gets them penalised. They will learn about its dick moves, and complain to Comcast to make them remove these fees so they can access Netflix, which they have already paid for access to.

Jtsummers 12y ago
Netflix would lose, they have competitors and Comcast (and potentially, in the future, ComcastTWC) has too many customers. Increasing the costs to all of them would result in the loss of too many customers before the dust settled.

snambi 12y ago

why there are no last mile providers like comcast and ATT?

jagger27 12y ago
Local governments grant monopolies to Comcast or AT&T.
- Jtsummers 12y ago
  And some state governments force it on the local governments (see NC and TWC versus Wilson, NC).

timr 12y ago

People keep claiming that there should be a "free market" for bandwidth...but then they say that the ISPs should have to absorb the costs of peering (which can be significant -- the hardware isn't free) without passing the costs on to anyone. The backbone providers complain when the cost is passed to them; the consumers scream bloody murder when the costs are passed to them in the form of a bill. Obviously, there's no free market in the status quo: we (consumers) basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure. There's an economic disconnect, and legislating that it shouldn't exist seems worse than futile. I say: pass the costs on to the consumer, and break down the monopolies on last-mile cable service. If the cable companies had to compete for subscribers, they could still pass on the costs of improving their infrastructure, but they'd have to compete with everyone else to do it. In other words, the problem here isn't "net neutrality" -- it's that we've got a monopoly at the last mile that we need to destroy.

Moral_ 12y ago
>which can be significant -- the hardware isn't free) without passing the costs on to anyone. I'm not trying to be rube but have you seen how much comcast/verizion/TWC customers pay? It's more than enough to string another switch and fiber line at a route point.
gph 12y ago
>basically expect to pay a low, ever-declining price for bandwidth, while someone else eats the costs of a growing network infrastructure Are we though? Our bills have only been increasing. The amount of money they put into infrastructure has decreased[1]. The Telecoms did have to put a lot of initial investment into infrastructure, but that was largely from laying the wires. Most of them are already starting to see positive returns on those investments [2]. They already have the infrastructure to support traffic to consumers in the last mile. The peering infrastructure being upgraded shouldn't cost them even 1/10th as much. So why should they get to pass their peering infrastructure costs on to the content providers? They should pass it on to the consumer if they must, after all it's us trying to get the content. But really they are going to be making such a large profit off us over the coming decades it's ridiculous for them to cry poor. Net neutrality is fairly meaningless when it comes to the last mile. I doubt hardly anyone is saturating their last mile worth of bandwidth. I regularly have three streams going in my household and it doesn't saturate my bandwidth. That part isn't what needs upgrading. [1]http://www.vox.com/2014/5/12/5711082/big-cable-says-broadband-investment-is-flourishing-but-their-own-data http://www.vox.com/2014/5/12/5711082/big-cable-says-broadban... [2]http://dailycaller.com/2013/02/15/does-cable-really-have-a-97-profit-margin/ http://dailycaller.com/2013/02/15/does-cable-really-have-a-9...

fragsworth 12y ago

The proposed solution is at the bottom of the article (which is why everyone seems to have ignored it): > The solution to this problem is simple: peering at the original NSFnet exchange points should be forever free and if one participant starts to consistently clip data and doesn’t do anything about it, they should be thrown out of the exchange point. I do have a couple questions though - who is in charge of the original NSFnet exchange points, and do they have this authority?

jamesbrownuhh 12y ago

The UK experience is, broadly speaking, that any ISP who is sufficiently tall to have the appropriate interconnects can offer a service to a customer via the incumbent's last-mile infrastructure. (This is for telephone-based ADSL broadband - the UK's only cable operator is not bound by this.) But, furthermore, competitor ISPs are enabled to install their own equipment and backhaul directly into local exchanges, known as LLU - local loop unbundling. LLU allows competitor companies to provide just your broadband, or your voice telephone service, or both. There is one further step, whereby the prices of the incumbent monopoly are regulated in areas where no competition exists. Ironically this works in the opposite way to how you'd think, as it forces the incumbent NOT to offer their lowest prices in that market - the intention being to make monopoly areas prime targets for competition and to ensure that potential competitors aren't scared out of the area by predatory pricing. It's an odd system with good and bad on both sides, but it seems a lot better than being stuck with a single source of Internet access.

jrochkind1 12y ago

You would think, okay, if Comcast is terrible at maintaining sufficient peering for it's customers needs -- and if the OP proposal to throw Comcast out of peering exchange points happened, that would certainly lead to increased terribleness for it's customers -- then eventually it's customers would choose a different ISP. The market would solve it. The problem is that in many many markets, Comcast (or another ISP) are pretty much the only choice. Customers don't have another option, no matter how much Comcast underfunds it's peering infrastructure or gets thrown out of peering exchange points. So what is the consequence to Comcat for underfunding? What is the consequence to Comcast for even such a disastrous outcome as getting kicked out of the peering exchange point? Not a lot. I'm not sure what the solution is, but 'regulate them as a common carrier' is certainly part of it, since they are a monopoly, and the common carrier regulatory regime was invented for exactly such a monopoly.

sadris 12y ago
> Comcast (or another ISP) are pretty much the only choice. Customers don't have another option If an entire town is angry, their city councilors will very quickly utilize eminent domain on the fiber in the ground and sell it to a competitor.
- kej 12y ago
  But not before the ISP lobbies the state legislature to decree that cities can't annex fiber.

guardiangod 12y ago

I don't know why everyone is up in arm over this. Here is reverse thinking and a perfect oppoturnity for everyone. The current situation is that Comcast doesn't have the equipment/resources to handle extra internet traffic at its peers. Most people want Comcast to buy more stuff to handle it, why don't we think the opposite way- get Comcast to decrease its amount of traffic? If we can get Comcast to consume less traffic, they wouldn't have to complain to other peers about load asymmetry. The best way to decrease traffic? Make Comcast has less customers. Why does Comcast has so many customers, even though their resources cannot handle it? Because they have a government mandated monopoly in the last mile, so they are forced to have more customers than what they can handle. We can come to a conclusion that last-mile monopoly -> network congestion -> forcing L3 to pay for peer. If Comcast has to compete with other ISPs for last miles, the traffic load would shift from 1 single entity (Comcast) to 10+ smaller ISPs. In such case the traffic load problem would not exist. Another solution is to breakup Comcast. See? This is a perfect opportunity. Comcast can has its multi-tier network, but at the price of the last mile monoploy. After all, if they want to have the right to choose peers, we customers should also have the right to choose ISPs.

cobookman 12y ago

I've previously interned at one of the mentioned Cable Companies, and I see both sides. The solution is to make it 'capitalistic'. Change all of our internet contracts from Unlimited (up to 'x' GB/month), to a simple $/gb cost. It would be in the ISPs best interest to provide their customers the fastest internet connection as possible. E.g, if a customer can stream a 4k video vs SD then the ISP would make more money per unit time. Think of it this way, if comcast charges $0.25/GB, and a netflix SD show is say 1GB and HD is 4GB, then comcast grosses $1 for HD and $.25 for SD for the same customer streaming request. Over time its likely the price per GB would decrease, just like it has for cellular. On a more evil side, this would also stop chord cutters. Pirating content is no longer 'free', and Netflix would cost significantly more than $10/month ($10/month + 'x'GB * $/GB). As for what rates to expect, if comcast charges in ATL $30-55 for 300GB, that'd be about $.10/GB to $.20/GB. As for speed tiers in a $/gb system, your guess is as good as mine.

seehafer 12y ago
How does changing plans from unlimited to metered data make them more 'capitalistic'? An example of making things more capitalistic would be to make illegal the municipal contracts granting monopolies on last mile service.
- cobookman 12y ago
  In an unlimited system, a customer which uses 10GB/month grosses the company the same amount as a customer who uses 1TB/month. Going to a metered system, people who use more bandwidth are charged more. Technically ISPs don't pay for bandwidth, but for a 'x' GBps link. But our electric companies do the same, they pay for 'x' Kw/h (a power plant can only output so much electricity per hour). >...make illegal the municipal contracts granting monopolies on last mile service. Thats a very hard peice of legislation to pass. I'm not even sure if Google Fiber would agree to this. Even if this legislation passed, would that fix our net neutrality rules? Moving to a metered system gets rid of most of the ISPs arguments against net neutrality....all of them except having a company pay for your bandwidth.
  - 12y ago

rossjudson 12y ago

Level3 should drop the same percentage of outbound packets from Comcast, that Comcast drops on the inbound. If every tier 1 did, Comcast's internet service wouldn't look all that good any more, would it?

jvdh 12y ago

Just for scale, backbone links these days are not 10 gigabits/sec, more in the order of 40-100 gigabits/sec. The Amsterdam Internet Exchange is the largest and most important exchange in Europe, and it's peak traffic each day reaches 3 terabits/sec.

mokus 12y ago

> Nobody paid anybody for the service because it was assumed to be symmetrical: as many bits were going in one direction as in the other so any transaction fees would be a wash. The justification for peering is not equal traffic, it's equal value - my customer wants to communicate with your customer. Regardless of the direction of traffic, the traffic is equally valuable to both of us because the traffic is the primary thing our customers are paying us for. Unless, of course, I can get you to pay me for it anyway because of some unrelated advantage such as the fact that your customers can leave you more easily than mine can leave me. Comcast and others are attempting to leverage exactly that - in many regions they have no viable competition whereas Netflix and L3 are much more replaceable in their respective markets. This is a prime example of abuse of a monopoly.

Rezo 12y ago

The extreme download vs upload traffic asymmetry between Comcast and L3/Netflix has been mentioned several times as a straw man argument for why Comcast is justified in charging Netflix directly. Maybe Netflix could find some creative uses for all that idle viewer upload capacity to reduce the deficit ;) - Have every Netflix client cache and serve chunks of the most popular streams P2P-style. You could have a DHT algorithm for discovering chunks or have Netflix's own servers orchestrate peer discovery in a clever way, for example by only connecting Comcast customers to peers physically outside of Comcast's own network. This would reduce Netflix's downstream traffic and increase viewer uploads. - Introduce the Netflix-Feline-Image-KeepAlive-Protocol, whereby every Netflix client on detecting a Comcast network uploads a 5MB PNG of a cat to Netflix's servers over and over again while you're watching a video. Strictly for connection quality control purposes of course.

tom_jones 12y ago

Along these lines, can someone ask whether net neutrality ever existed at all? Akamai and F5 have been helping big corporations like Disney circumvent internet bottlenecks for over a decade now. Those who have had the money have managed to purchase faster delivery schemes for over a decade. Could it be, then, that telecommunications companies are consolidating so that they can extort money not from the small guys, but from the big guys? Are Hulu, Netflix and others willingly submitting to the extortion because they see no other way out? To be sure, the telecommunications industry is in desperate need of regulation because providing good service at a reasonable price for a reasonable profit is not good enough for them.

eb0la 12y ago

The problem is not to peer or not to peer. The problem is WHERE to peer. I work for a european ISP and the problem we have is the location of the peering. Big content providers will happilly peer with you in, say, Palo Alto or Miami; but they will refuse to add a peering connection in Europe. Why? because today the problem is about WHO pays the Intercontinental route (which limited and is expensive bandwidth). Level3 is known in the industry as a pioneer for bit-mile-peering agreements. This means you have to sample the origin and destination of the IP packets and make some calculations to know how many miles the packet has traveled and pay / get paid if someone dumps long haul traffic to a peer. Getting to this is complicated with current tecnhology and many companies are refusing to peer with Level3 because they don't know what will happen with their business with bit-mile-peering agreements.

Eye_of_Mordor 12y ago

Lack of competition all around - just break up the big boys and everything will be fine...

GregFoley 12y ago

The problem would disappear if ISPs used metered pricing. Why do we have unlimited commercial broadband?