> in this case, since Level3 is pushing more data, they would be the ones paying
No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.
They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.
If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.
This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.
Well put.
Yours is probably the clearest explanation I've seen of why this all feels so icky.
As I like to say it, saying that Neflix uses a lot of bandwidth is like saying that the Hetch Hetchy uses a lot of water pipes.
You still haven't refuted the point. This isn't about the relationship between Verizon and its customers, at all.
The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon.
Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.
I know the answer to this one! We need these guys to setup at the other end of a Level3 link, and then Verizon customers can push them one byte for each byte they get from Netflix:
http://devnull-as-a-service.com http://devnull-as-a-service.com
Imbalance fixed, but I don't think that is what Verizon wants.
Seriously though: what if Netflix worked with Level3 to set up an efficient Level3-owned /dev/null server in the carrier hotel AND have every Netflix client that connects over Verizon upload random bits to that IP address whenever buffering is complete or the movie is paused? Then the rates would balance out, and Verizon would no longer have a contractual excuse to NOT hook up those cables to the Level3 endpoint. The only way for Verizon to mitigate this would be to deny their end users the upstream bandwidth they've paid for but are not using (cue user outcry), or to smear Netflix even more so than they're already doing (cue more chances for the PR to backfire). Verizon would hate Level3 for this, but they already do, and they couldn't do anything about it without denying their end users Netflix access altogether!
Of course, since the NSA would need to listen in on all that new random-bit traffic (since Netflix is obviously the new Skype in the terror playbook), taxes would need to rise significantly in order to pay for more NSA bulk-data-collection servers. But that's just the cost of living in America...
Then how would you explain Verizon's refusal to host Netflix content inside its network that's under-utilized?
Seems to me it's a win-win for everyone, including poor border routers. Except Verizon then wouldn't make an extra penny from that efficiency, hmm?
It is entirely about the relationship between Verizon and their customers. Verizon's customers want these interlinks, they just don't know it. This whole thing started when Netflix posted a message to their customers explaining it. Verizon is trying to defend themselves against the complaints of their customers.
ISPs charge me primarily to download. Literally I can't upload as much as I download. My uploads are contractually limited to 20% of my downloads.
So does that mean there's a downloader-pays rule?
If so that means that Verizon should be paying Level3 because it consumes 3x the bandwidth that it provides.
Verizon's needs Level3 3x as badly as Level3 needs Verizon on a bit-by-bit basis. So why isn't Verizon paying Level3 for the imbalance?
There are actually two sides to the ISP game. There's the consumer ISP side and there's the commercial ISP side.
Consumer ISPs are in business to get their customers hooked up to as much of the internet as possible to enable them to consume things. And they charge their customers (normal folk like us) for this service.
Commercial ISPs are in business to get their customers hooked up to as much of the internet as possible to provide services. And they charge their customers (big companies) for it.
When commerial ISPs meet consumer ISPs typically they say "let's trade traffic for free so that way we can both keep our money and not worry about accounting"
What's happened is that a consumer ISP is trying to charge more than just it's last-mile normal customers for data that they use and which it is ostensibly contractually obliged to provide to them without further fees.
What Verizon is saying is "look they need to pay us to upgrade this link" which on the surface doesn't sound too bad. It kinda makes sense. If I used more bandwidth they would charge me more. But the thing is that I am already paying them to make sure that the connection to the internet is uncongested. They are deliberately allowing it to get congested to try and bill both sides of the transit.
This. Completely this.
They may have a peer balancing agreement, which is not being held up, but they're also still being payed by their customers to provide access to content outside of their own network and Verizon is enforcing a limitation from another agreement (the peering agreement) that is affecting their ability to provide the content their customers want to have access to.
You can't have it both ways. Or... I guess you can, as Verizon has shown.
Commercial ISPs vs. Consumer ISPs is a useful way to frame the issue. From an economic perspective, the big difference is that the Consumer ISPs generally have a monopoly over the Consumer endpoints, while the Commercial ISPs do not have a monopoly over Commercial endpoints and must negotiate a competitive market.
This obviously gives the Consumer ISPs pricing power in all of their business dealings (including any peering contracts that they force the Commercial ISPs to sign). The fact that Verizon owns RedBox (a Netflix competitor) & sells TV packages is just further motivation for hampering the interconnection.
Verizon is charging me for access to the internet -- then charging the internet for access to me. That is double-billing, and it's egregious.
Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity. This is the same company that over-subscribes their network on purpose, as most ISP's do. They will come into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes that not everyone uses the connection at the same time, nor for long duration.
Netflix has actually addressed this in one of their blog posts: http://blog.netflix.com/2014/03/internet-tolls-and-case-for-strong-net.html http://blog.netflix.com/2014/03/internet-tolls-and-case-for-...
> But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs.
> in other words, moving to peer-to-peer content delivery
Verizon gave a very clear suggestion for Netflix: work with other providers as well.
Why should they have to do that? Level 3 has open ports now (so they claim). It doesn't matter which provider the traffic is coming from, either Verizon wants it or they don't (and apparently they don't, even though their core is no where close to saturated).
The balanced traffic requirement is bogus anyway. Its not as if Verizon offers their own video streaming service that I can subscribe to as a Comcast or AT&T customer. So they're never going to see balanced traffic in the age of video streaming.
I don't think in the history of the internet there has ever been a "balanced" traffic arrangement. The majority of people are always going to be requesting more than they are uploading.
As a residential subscriber this is contractually the case. ISPs do not allow the operation of servers in residential contracts and they are almost always provisioned with asymmetric speed. 50 down, 5 up, etc.
The concept of asymmetry as you're using it doesn't make sense in this situation.
- If you push traffic to my network it is because someone in my network requested it.
- Incoming and outgoing traffic are technically no different.
- It makes more sense to pay to receive traffic than get paid to receive traffic. That's actually what VZ customers pay for and that's why not many people complain about ADSL speeds.
If I were Level3 I would actively partner with VZ alternatives in the US, announce it to VZ customers so they can move away and then proceed to remove any peering with VZ. Enough bullshit already.
I have three questions:
Instead of double billing (i.e. subscriber + content providers) couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts? Claiming that Verizon charges Netflix at extra e.g. $1/subscriber/month to peer?
Would it be legal for Verizon to literally just cut off Netflix? Just return an access denied to their subscribers.
(3) I actually don't know off the top of my head.
(1) would certainly be doable, but it's something Verizon wants to avoid because in most markets they're in a duopoly (often with Xfinity, sometimes with a local provider). Much like two gas stations across the street from each other often having identical prices, Verizon doesn't want to be perceived as costing more for the same service as the alternative.
For (2), Netflix could almost certainly do that, but I assume they'd prefer to not pass any cost onto consumers (because, really, they're making a stand against this business practice by Verizon being fair play at all). Among other reasons, if it's seen that Netflix is willing to pay a "ransom," other carriers may try to increase their contracted costs.
Respectively, yes, yes in the absence of network neutrality, and yes in the absence of network neutrality.
We think of ISPs as selling us data, but the big ones have realized they can make more money as data market makers. This whole kerfuffle is just them trying to see if they can get away with it. Because they're effectively monopolies on the last mile, it's important that they fail here.
couldn't Verizon just increase their subscriber rates to cover their perceived increased costs?
Verizon already increases prices x% per year, so maybe. But for every price increase there are some number of customers who will cancel or downgrade to a lower plan. (Alternately, Verizon could decrease their profits.)
Could Netflix pay Verizon's ransom and then charge Verizon customers a "Verizon Tax" of sorts?
Sure, but some percentage of Netflix customers would cancel.
Would it be legal for Verizon to literally just cut off Netflix?
Considering that the FCC is debating Net Neutrality right now, that would be a very bad idea.
It doesn't take too much away from your point, but people do "push" data outside the context of an attack. IP is not a request/response protocol like HTTP.
Agreed, it's technically possible. In practice it almost never happens. And even when it does it might TECHNICALLY be a push but from a very high level perspective it might still be a pull.
As an example I use a website (like Facebook or something) which does live auto-updating. Under the hood it's something like WebSockets or SocketIO and the server does send datagrams to me without me specifically requesting those individual data packets. I couldn't request them, I didn't even know about them until they arrived!
But while that technically is "push" from a high level it's still "pull" in that Facebook (or whoever) doesn't just start sending them to me for giggles. It doesn't start happening until I visit their site, and it stops happening fairly shortly after I navigate away from it. Their computers don't unilaterally send packets to me. I request them by continuing to execute the javascript that their servers sent to me when I loaded the page. Once I stop executing it, the connection is closed and the messages stop.
PUSH and PULL don't really describe how it works.
Its the senders responsibility to deliver data to the local ISP of the recipient. It doesn't matter if the person asked for the data, you still have to pay to get it there.
The issue is that Verizon is both a national transit provider and a local ISP. If you just dump your data on the nearest Verizon peering port, you aren't fulfilling your end of the bargain. You are asking Verizon to do it for you, as a transit provider.
However, if you are delivering it to the last mile, you have "paid for it" it already.
The issue is where are these congestion issues occurring. At the last mile or at the edges of Verizon's transit network.
Verizon absolutely has the right to bill for providing transit.
The question then becomes does the contract Verizon have for peering with other transit providers allow them to charge the other transit customers for that? And if they do what's to stop every transit provider from charging every other transit provider's customers for the same thing? And then the all important question, SHOULD they be stopped from doing it? The current structure of the internet was built around this not happening that way, but is that the way it should be? I strongly believe that letting it happen would cause many other political and routing problems for the internet as a whole to the point where it'll end up being regulated against doing it that way anyway. It could also end up then as a zero sum game between all the transit providers ending "free" peering agreements.
It's a fair distinction. Here, I believe, the movie is going AWS Seattle to L3 to the front door of Verizon's consumer network in LA, and Netflix is paying for it all the way to the front door.
Is that accurate? Or no?
> That's not how the internet works. Nobody PUSHES data. People PULL data.
There's more to "the Internet" than just "the WWW". While HTTP could be considered a "pull data" protocol, there's plenty of application protocols which involve "pushing" data over TCP/IP transport.
Yes I'm aware. But the vast majority of traffic happens as a result of a request of some kind.
In other words Netflix doesn't start streaming data to me for the hell of it. Pandora doesn't stream songs to my computer unless I hit play. Songs don't get downloaded from iTunes unless I request them and pay for them.
This is in contrast to the mail that the USPS delivers to my door. I don't ask for most of it and I have to sort through it and throw all the bullshit away.
I guess what I mean is that most internet traffic is based on consent of some kind. While paper mail is not.
I might not specifically request any particular bits but the majority don't show up unannounced and unwanted.
> there's plenty of application protocols which involve "pushing" data over TCP/IP transpor
Right but by the time a TCP link is established you've already setup a session which can't be done unilaterally. That is akin to consent and quite possibly a request. You can't be "data-raped" because if you decline the TCP session that's the end of it. Someone can send a billion request to initiate TCP sessions but they can't initiate without your (or your computer's) consent.
Given the router/firewalls that basically everyone has between their modems and their computers it's getting very, very difficult to send data to residential ISP customers without their express or implied consent.
deleted 12y ago
[deleted]
1) Level3 has offered to split the cost with the Big5 ISP's of upgrading their infrastructure.[1]
* This would increase throughput and capacity for both Level3 and the consumer networks.
2) Netflix has offered free Open Connect Appliances to all major ISP's.[2]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all their customers. In-network traffic is almost free for the ISP's.
3) Netflix has offered to change their entire distribution model into a P2P model.[3]
* This would allow consumer networks to only download a video once, then serve it up locally within their network to all customers. In-network traffic is almost free for ISP's.
.
All of these options have been flat-out refused. The Big5 ISP's are purely after the money. There is no other compromise for them.
[1] http://blog.level3.com/global-connectivity/chicken-game-played-child-isps-internet/ http://blog.level3.com/global-connectivity/chicken-game-play...
[2] https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
[3] http://arstechnica.com/information-technology/2014/04/netflix-researching-large-scale-peer-to-peer-technology-for-streaming/ http://arstechnica.com/information-technology/2014/04/netfli...
From elsewhere on HN:
All of Netflix, in HD, on 24TB.
Netflix could solve the bandwidth problem by just mailing the whole library to customers. "Never underestimate the bandwidth of a station wagon full of magnetic tapes."
Netflix offer this as an option to any ISP that wants it, free.
They will put a server in a data center anywhere an ISP wants, and they'll manage the whole syncing thing so the ISP doesn't even know it's there. After that, all content comes from that cache.
To clarify Grecy's statement, netflix offers openconnect[1] for to ISPs as a caching mechanism, and the box contains about 108TB of disk capacity[2]. The boxes are sent to ISPs instead of customers though.
[1] https://www.netflix.com/openconnect https://www.netflix.com/openconnect
[2] https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
deleted 12y ago
[deleted]
> No. That's not how the internet works. Nobody PUSHES data. People PULL data.
To be pedantic curl -d test example.com would push data across the internet.
> If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
> Double billing for a single service is a neat trick if you can pull it off.
This is how every communications network has been ideally built from the beginning and this "trick" has been played out time and time again starting starting with the PSTN. It's not uncommon, but is also isn't ideal - unless you are the carrier.
>Not always. Backup is popular today - as I look at my simple home utilization I push about 30% of the ingress up to "somewhere".
You're equivocating on 'push' and 'pull'. Under OPs definition, a 'pull' is a customer's valid usage of the service, while a 'push' is something like unwanted or unrequested data transfer from a 3rd party. This definition is a little muddled when you talk about uploads but it should be easily understood. You uploading your backup data is you merely using the service you paid for, even though you're 'pushing' data somewhere.
Furthermore, it's not as though the backup company doesn't want the data and you're trying to cram it down their throats anyhow. Are you using more upload that download while doing the backup? Yes. Is the backup company's server sending ACKs to let your client know to continue sending data? Also yes.
>No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Uh, I don't think he meant it the way you took it. I think he knows how network traffic works.