8 ms·
Netflix Open Connect Content Delivery Network
- rmrfrmrf 12y agoI read that entire OpenConnect Deployment guide PDF, and I have to say that it looks like a really great, well-thought-out program. I'm impressed with how many moving parts there are and the time it must have taken to develop a system like this; they don't even touch on the details behind the Cloud Control Plane, and I'm sure that must have taken a while to develop. Netflix has said in the past that they only want the most skilled employees possible, and it certainly shows here.
- mey 12y agohttps://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/software https://www.netflix.com/openconnect/software It looks like they are heavily redundant boxes with the goal of reducing field maintenance, running FreeBSD as the OS.
- Alupis 12y ago> running FreeBSD as the OS. Presumably for ZFS
- deleted 12y ago[deleted]
- howeyc 12y agoActually it turns out they don't really have a need for most of the features of ZFS. So they don't use it. http://www.youtube.com/watch?v=FL5U4wr86L4 http://www.youtube.com/watch?v=FL5U4wr86L4
- Alupis 12y agoHmm, do you know which spot in the video they discuss that (it's a long video)? Seeing that they are using just controller cards instead of raid cards, and FreeBSD, it seems even if they were not using specific features of ZFS, the performance and other automatic things ZFS does to provide superior (compared to most (all?) other filesystems) data integrity and protection... they would be using it by default...
- Thaxll 12y agoPerformance is not a strength of ZFS.
- howeyc 12y agoWhy FreeBSD - 18:30 Why no ZFS - 21:30
- justincormack 12y agoIf a disk does they just stop using it and continue with less capacity.
- Duckpaddle2 12y agoVery interesting to see what hard drives they "blessed".
- Alupis 12y agoGoogle uses standard off-the-shelf sata drives too. Too many disk failures to be worth spending money on the expensive "enterprise" drives when a standard SATA drive will due the job just fine (especially in arrays like this -- plenty of spindles to get performance)
- danudey 12y agoBackblaze also uses off-the-shelf HDDs (including, when HDDs were in short supply after the flood in Thailand, buying USB drives and tearing the case off). They publish a lot of reliability numbers on drives as well (which match up quite well with my own experiences), as well as cost/GB numbers.
- Alupis 12y agoI remember that! I think "This Developer's Life" or a similar podcast interviewed them about trying to game the walmart hdd rationing (2 per customer) and stuff.
- nacs 12y agoThe hard drives used are listed here: https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware Hitachi Deskstar 5K3000 3TB Seagate Barracuda 7200.14 3TB
- ihsw 12y ago> The following system was developed and first deployed at the end of 2011. This is very out of date. Their latest OpenConnect appliances use SSD drives, have higher capacity, and use less power. They're heavier, too.
- blerk 12y ago> For the operating system, we use FreeBSD version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. ...or due to the less restrictive licensing, it's ok netflix everybody does it...
- Alupis 12y agoFreeBSD is pretty standard for most storage servers, be it CDN's, file hosting, etc. When they say "BSD powers the internet", it's not really that much of an exaggeration.
- wmf 12y agoAFAIK Netflix contributes back all their changes to FreeBSD, so I don't see how the license matters. My impression is that companies that don't like the GPL either deal with it (Android) or just violate it (almost all embedded devices). Nobody is actually willing to switch to BSD for licensing reasons.
- blerk 12y agoSony switched over from Linux (ps3) to kfreebsd (ps4) due to licensing. Sure they (netflix) contribute back to FreeBSD, I don't doubt it, but don't act like licensing doesn't have anything to do with it.
- curun1r 12y agoThe one thing missing from all the articles about OpenConnect is any way for Netflix subscribers to see which ISPs are participating. I'd like to know if my ISP participates and, if not, be able to call in to suggest they do. If they still don't join in a reasonable time frame, then I'd at least have the option to switch providers to someone that participates. A program like this will be more successful if they allow for consumers to pressure ISPs to participate.
- deleted 12y ago[deleted]
- Alupis 12y agoUnfortunately, as we've discovered from the whole net neutrality debate -- netflix has offered these boxes for free to all major ISP's... yet the Big5 ISP's have all turned them down.
- Thaxll 12y agoYes because those box aren't sufficient for 1M+ sub. You're better off using peering rather than cache boxes in your network if you're a large ISP.
- Alupis 12y agoWell, the boxes have multiple 10Gb ports on them -- so they can be as fast as the ISP will allow it to be on their network. And, Netflix will provide (free of charge) as many OpenConnect devices an ISP needs to serve all of their mutual customers. Also, Netflix offered to change their distribution model into a P2P model as well -- the Big5 ISP's said no to that too.
- sp332 12y agoThat doesn't mean the hard drives can serve multiple 10Gb at once. And why would ISPs give them space, power, and cooling for free when every other colo'd box (including CDNs like Akamai) pay for that?
- swasheck 12y agoit seems to me that calling this an "internet fast lane" is a false equivalency. my understanding was that the internet fast lane concept was something that users would pay more to use, which leaves the isps free to render the "regular lane" unusable so as to bump the pricing. additionally, there was the whole notion of the "bundles" of websites available at different cost tiering. openconnect appears, at least to me, like it's a collaboration between providers of different services to enhance the value for their shared customers. edit: directed at both mmalone and condiment
- deleted 12y ago[deleted]
- Alupis 12y agoThe entire point of OpenConnect was to help ISP's reduce the inbound traffic burden into their networks. Normally, Netflix has a CDN and when a customer views a video, that CDN serves the content --> which comes in through the ISP's providers --> costing the ISP bandwidth. With OpenConnect, Netflix offers to give any major ISP that requests it, a free (or multiple free) OpenConnect devices. They are hands-off zero maintenance for the ISP, and they put them in the ISP's data centers. This means, when a Netflix customers goes to view a video, it gets streamed off the OpenConnect appliance which is already inside the ISP's network -- and in-network traffic costs the ISP's almost nothing. So, with OpenConnect, the ISP downloads a video once, then serves it up locally. Netflix customers get lower latency videos, and sometimes even higher HD content (overall better service), and ISP's get to reduce their traffic burden significantly, which saves them money. This should be/is a win-win for everyone involved, the ISP, the mutual customer, and Netflix. (the problem as we've seen through the net neutrality debate, the Big5 ISP's here in the US have all refused the OpenConnect appliances)
- grecy 12y agoI work at an ISP with local caches from a few of the big players. We've been contemplating getting a NetFlix OpenConnect device, but as yet it doesn't make sense for us. The way it works is that once per day, it will sync the ENTIRE NetFlix catalogue to the OpenConnect device, which is an enormous amount of traffic (for us). Even if it's a movie or TV series none of our customers have ever watched, it still gets synced. Because we only have ~25k internet subscribers, it would actually be more traffic to sync the OpenConnect device daily than we're currently pulling from NetFlix. Obviously this would be different with vastly more customers, and we're watching the numbers as more and more of our internet customers subscribe to NetFlix. We wish NetFlix's caching worked like some of the other big players, in that "popular" content is cached, as is anything watched by someone. If an asset is not popular and never watched, it will never move into our network.
- 12y ago
- mmalone 12y agoIf you're a net neutrality advocate you should seriously consider whether this sort of peering arrangement should be tolerated. This kind of peering is a de-facto QoS arrangement that gives Netflix a performance and cost advantage over other video providers that aren't big enough or rich enough to justify and pay for direct peering with ISPs. It also gives bigger ISPs (that do more than 2Gb/s of traffic to Netflix and agree to peer, apparently) the same advantage over their smaller competitors. Isn't that exactly what we're trying to avoid with net neutrality? This is not a new practice and I'm surprised it doesn't come up more frequently in the net neutrality debate. It's been going on for years -- broadcast.com did it, YouTube and Google do it, and I'm sure others do as well. We've never really had net neutrality -- peering agreements have been happening forever in shady back room deals at NANOG. At least the more recent shenanigans are happening out in the open.
- JoshTriplett 12y agoAlternatively, if you're a network neutrality advocate, you should consider whether this is really the kind of thing you're advocating against. It's one thing to complain about ISPs intentionally throttling or dropping traffic, or shaking down services to get paid twice for the same traffic. It's something else entirely to complain about services going out of their way to establish CDNs near their customers and reduce overall network load.
- mmalone 12y agoComcast is throttling / dropping traffic because they want Netflix to pay for bandwidth. You're right that they'd be getting paid twice for the same traffic, but ISPs already do this -- if Comcast and Netflix are connected via Level 3 they both have to pay Level 3 for the same traffic. What if Comcast offered to host Open Connect devices for a fee -- would that be a violation of net neutrality? To me that just seems like the free market doing what free markets do (I'm ignoring for a moment that Comcast is basically a monopoly which is a legitimate problem). What if Comcast throttled everyone _except_ Netflix. Is that really different than Comcast allowing _only_ Netflix to peer?
- Alupis 12y ago
- condiment 12y agoFor all the hand-wringing about ISPs creating an "internet fast lane", OpenConnect sure seems like a step in that direction. If Netflix is allowed to colocate their server in an ISP's facilities (effectively eliminating the cost of their egress bandwidth), how does that not give them an unfair advantage against other internet video operators who lack the leverage necessary to broker this kind of deal? OpenConnect isn't the first attempt at this sort of business model, either. Back in the 90s, Akami tried this same thing with their "FreeFlow ISP" service model - putting their servers in the ISP's point of presence in order to eliminate their bandwidth costs. Then, as now, major ISPs refused.
- deleted 12y ago[deleted]
- StavrosK 12y agoCan someone explain what this is? I don't see that mentioned anywhere.
- whoopdedo 12y agoWhat's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect? The way that satellite TV networks charge a subscriber fee to the cable operators. Netflix has stated that they want to become like HBO. I don't imagine they're unfamiliar with how HBO become so successful.
- wmf 12y agoWhat's to stop Netflix from later on asking the ISPs to pay for the cost of OpenConnect? The fact that ISPs will unplug it five seconds later. The way that satellite TV networks charge a subscriber fee to the cable operators. HBO only charges each customer once (via the cable company). If Netflix tried to double-dip by charging the customer directly and also charging the ISP they'd get rightly flamed.
- whoopdedo 12y ago> The fact that ISPs will unplug it five seconds later. At the risk of angering the many people who watch Netflix. I don't get the double-dipping argument either. We already have that with television, newspapers, and radio being paid by advertisers yet still charging a subscription to viewers.
- Touche 12y agoThey already are paying. These servers require power and space.
- 13throwaway 12y agoI wonder what measures they have in place to protect against someone cracking this open and dumping the entire video library.
- nodata 12y agoFunny name: is it open to other CDNs too?
- rdl 12y agoI wish I could buy these custom chassis; I didn't see a link in the article.
- mbell 12y agoI would think in low volume a supermicro chassis would be more cost effective, they have many 4U models that support 36+ 3.5" drives.